https://pantsbuild.org/ logo
#general
Title
# general
q

quiet-dog-11469

02/28/2023, 8:45 AM
I’m having trouble getting the git auth to work with pants for installing python packages from a private github repo. It works fine using SSH but not all developers have SSH access set up and want it to be able to work with a github token. More details in 🧵
👋 I’m new to
pants
and giving it a test to see if it’ll work for our organisation. When generating the lockfiles or installing I get the standard pip 128 error that it can’t
git clone
and there’s no prompt to enter github username and password. I’ve tried the following: 1.
git credential store
to store the token 2. Changing the url in the git config to use the token directly:
git config --global url."https://${GITHUB_TOKEN}@github.com/".insteadOf "<https://github.com/>"
Both of these methods work fine for
git clone …
or
pip install …
but not with
pants
😕 How does
pants
interact with
pip
and
git
? Is there a setting I can change to allow it to use either of these, some thing else entirely or supply a token? For note the lines in the
requirements.txt
look like
Copy code
PACKAGE_NAME@ git+<https://github.com/ORG/REPO.git@TAG>
r

refined-addition-53644

02/28/2023, 9:49 AM
q

quiet-dog-11469

02/28/2023, 9:51 AM
Hi thanks. The packages I want to install sadly aren’t host on a custom pypi repo but are installed directly through github from a private repo.
pants
does something to
git
or the
git config
for the
git clone
command so the users
git
config isn’t used. I just have no idea what 🤷‍♂️
r

refined-addition-53644

02/28/2023, 9:55 AM
What happens if you use
Copy code
PACKAGE_NAME@ git+https://${GITHUB_TOKEN}@github.com/ORG/REPO.git@TAG
I feel like pants will have to store these credentials somewhere inside lockfile which doesn’t sound great.
😬 2
😕 1
Easy way is to force everyone to use SSH 🙂
👍 1
q

quiet-dog-11469

02/28/2023, 10:15 AM
force everyone to use SSH
Sadly that would be harder as it would require changes to our remote dev environments 🙃 This does work
Copy code
PACKAGE_NAME@ git+https://${GITHUB_TOKEN}@github.com/ORG/REPO.git@TAG
but does store the
GITHUB_TOKEN
in the
lockfile
😢
Do you know how
pants
interacts with
pip
/
git
. Does it sandbox them from the user’s configs?
r

refined-addition-53644

02/28/2023, 10:16 AM
Everything is hermetic in pants. Pants actually uses PEX which uses pip for git.
👍 1
Let’s wait for someone from core pants team to further help you. This is now beyond my knowledge. If you want dig inside the source code, here is the one for generating the lockfile https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/lockfile.py
🤗 1
q

quiet-dog-11469

02/28/2023, 11:02 AM
Strangely (at least to me as a novice)
pex -r requirements.txt
works with my global git config (with no
GITHUB_TOKEN
in my
requirements.txt
) I’m assuming it all comes down to how tight is the sandbox and what should or shouldn’t be included in the sandbox from global or system configuration. I’m assuming as the global configuration is stored in the home directory (
~/.gitconfig
) then this is sealed from the sandbox. Also the
git
credential store is stored there too (
~/.git-credentials
)
r

refined-addition-53644

02/28/2023, 11:04 AM
Did you try generating the lock? I think it’s bit different command.
Copy code
pex --lock
If that works without injecting any credentials, then I suppose pants is the one blocking it.
Note that I am no pex expert, but my other guess would be pex just fetches the local copy/cache of these dependencies. So I would completely uninstall it and then try generating the lockfile?
q

quiet-dog-11469

02/28/2023, 11:23 AM
Yes generating the lock directly with
pex3 lock create
works so it is the pants sandboxing blocking it
c

curved-television-6568

02/28/2023, 1:15 PM
so likely identifying which env var(s) are needed could solve this. I would take a banter on
HOME
first.. https://www.pantsbuild.org/docs/reference-subprocess-environment
e

enough-analyst-54434

02/28/2023, 3:20 PM
Yeah, definitely
HOME
. Both
~/.gitconfig
and `~/.netrc`can only be found if you know the home dir.
q

quiet-dog-11469

02/28/2023, 3:39 PM
Works like a charm thanks . Is it worth updating or adding more documentation to add this or it more of a me issue 😅 ? Potentially adding another paragraph or two to version control requirements?
e

enough-analyst-54434

02/28/2023, 3:49 PM
We can whac-a-mole, but ideally the big point we'd get across early is that Pants does not allow things it runs to see your environment variables. Knowing that, and knowing how to punch environment variable holes equips you to solve many varying problems transitioning to Pants.
Same technique works for Docker auth and ... a few other things presently, more in the future.
@quiet-dog-11469 if you want to whac the mole in the name of incremental progress, definitely submit a PR: https://github.com/pantsbuild/pants/blob/main/docs/markdown/Python/python/python-third-party-dependencies.md#version-control-requirements It's great to get the wording fresh off the painful experience, it's more likely to really address the confusion frame.
q

quiet-dog-11469

02/28/2023, 4:04 PM
I’m all for not playing whac-a-mole, although sometimes you can get further than you expect with it. I had another skim & search of the docs and I can’t see this called out anywhere (explicitly at least). However, I’m not sure the best place to put it 1. How does pants work is too much of a high level “advert” 2. Troubleshooting / common issues feels like it’s been left a little bit too late. Although it would be a good place to start Is there anywhere else obvious I’m missing or have I been a poor reader?
e

enough-analyst-54434

02/28/2023, 4:06 PM
I am not the person to ask here. I tend to find this sort of thing is viewed differently by nearly every person. Obvious place for one, totally missed by another. Maybe someone else can chime in.
c

curved-television-6568

02/28/2023, 4:15 PM
I think calling this out early, such as in https://www.pantsbuild.org/docs/how-does-pants-work#hermetic-execution could make sense. That hermetic topic is very brief for such a fundamental aspect of the system and could benefit from a few more words on what the implications are, such as this example for instance. my cents 😉
👍 1
q

quiet-dog-11469

02/28/2023, 4:52 PM
I’d added a tiny PR
🙏 1
6 Views