I’m having trouble getting the git auth to work wi...
# general
q
I’m having trouble getting the git auth to work with pants for installing python packages from a private github repo. It works fine using SSH but not all developers have SSH access set up and want it to be able to work with a github token. More details in 🧵
👋 I’m new to
pants
and giving it a test to see if it’ll work for our organisation. When generating the lockfiles or installing I get the standard pip 128 error that it can’t
git clone
and there’s no prompt to enter github username and password. I’ve tried the following: 1.
git credential store
to store the token 2. Changing the url in the git config to use the token directly:
git config --global url."https://${GITHUB_TOKEN}@github.com/".insteadOf "<https://github.com/>"
Both of these methods work fine for
git clone …
or
pip install …
but not with
pants
😕 How does
pants
interact with
pip
and
git
? Is there a setting I can change to allow it to use either of these, some thing else entirely or supply a token? For note the lines in the
requirements.txt
look like
Copy code
PACKAGE_NAME@ git+<https://github.com/ORG/REPO.git@TAG>
r
q
Hi thanks. The packages I want to install sadly aren’t host on a custom pypi repo but are installed directly through github from a private repo.
pants
does something to
git
or the
git config
for the
git clone
command so the users
git
config isn’t used. I just have no idea what 🤷‍♂️
r
What happens if you use
Copy code
PACKAGE_NAME@ git+https://${GITHUB_TOKEN}@github.com/ORG/REPO.git@TAG
I feel like pants will have to store these credentials somewhere inside lockfile which doesn’t sound great.
😬 2
😕 1
Easy way is to force everyone to use SSH 🙂
👍 1
q
force everyone to use SSH
Sadly that would be harder as it would require changes to our remote dev environments 🙃 This does work
Copy code
PACKAGE_NAME@ git+https://${GITHUB_TOKEN}@github.com/ORG/REPO.git@TAG
but does store the
GITHUB_TOKEN
in the
lockfile
😢
Do you know how
pants
interacts with
pip
/
git
. Does it sandbox them from the user’s configs?
r
Everything is hermetic in pants. Pants actually uses PEX which uses pip for git.
👍 1
Let’s wait for someone from core pants team to further help you. This is now beyond my knowledge. If you want dig inside the source code, here is the one for generating the lockfile https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/lockfile.py
🤗 1
q
Strangely (at least to me as a novice)
pex -r requirements.txt
works with my global git config (with no
GITHUB_TOKEN
in my
requirements.txt
) I’m assuming it all comes down to how tight is the sandbox and what should or shouldn’t be included in the sandbox from global or system configuration. I’m assuming as the global configuration is stored in the home directory (
~/.gitconfig
) then this is sealed from the sandbox. Also the
git
credential store is stored there too (
~/.git-credentials
)
r
Did you try generating the lock? I think it’s bit different command.
Copy code
pex --lock
If that works without injecting any credentials, then I suppose pants is the one blocking it.
Note that I am no pex expert, but my other guess would be pex just fetches the local copy/cache of these dependencies. So I would completely uninstall it and then try generating the lockfile?
q
Yes generating the lock directly with
pex3 lock create
works so it is the pants sandboxing blocking it
c
so likely identifying which env var(s) are needed could solve this. I would take a banter on
HOME
first.. https://www.pantsbuild.org/docs/reference-subprocess-environment
e
Yeah, definitely
HOME
. Both
~/.gitconfig
and `~/.netrc`can only be found if you know the home dir.
q
Works like a charm thanks . Is it worth updating or adding more documentation to add this or it more of a me issue 😅 ? Potentially adding another paragraph or two to version control requirements?
e
We can whac-a-mole, but ideally the big point we'd get across early is that Pants does not allow things it runs to see your environment variables. Knowing that, and knowing how to punch environment variable holes equips you to solve many varying problems transitioning to Pants.
Same technique works for Docker auth and ... a few other things presently, more in the future.
@quiet-dog-11469 if you want to whac the mole in the name of incremental progress, definitely submit a PR: https://github.com/pantsbuild/pants/blob/main/docs/markdown/Python/python/python-third-party-dependencies.md#version-control-requirements It's great to get the wording fresh off the painful experience, it's more likely to really address the confusion frame.
q
I’m all for not playing whac-a-mole, although sometimes you can get further than you expect with it. I had another skim & search of the docs and I can’t see this called out anywhere (explicitly at least). However, I’m not sure the best place to put it 1. How does pants work is too much of a high level “advert” 2. Troubleshooting / common issues feels like it’s been left a little bit too late. Although it would be a good place to start Is there anywhere else obvious I’m missing or have I been a poor reader?
e
I am not the person to ask here. I tend to find this sort of thing is viewed differently by nearly every person. Obvious place for one, totally missed by another. Maybe someone else can chime in.
c
I think calling this out early, such as in https://www.pantsbuild.org/docs/how-does-pants-work#hermetic-execution could make sense. That hermetic topic is very brief for such a fundamental aspect of the system and could benefit from a few more words on what the implications are, such as this example for instance. my cents 😉
👍 1
q
I’d added a tiny PR
🙏 1