Sigh, another probably obvious thing I'm missing. Is there a trick to using private repositories in python requirements and passing authorization? I have a

mytarget@ <git+ssh://git>@github.com:/myorg/myrepo.git@myhash

line in a

requirements.txt

file, but doing

./pants test ...

results in a

ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com:/myorg/myrepo.git' /tmp/process-executionkhgvsl/.tmp/pip-download-1qlw9fjv/mytarget Check the logs for full command output.

...but copying/pasting the

git clone

command works just fine in the same shell (with ssh-agent); it's just not being passed to pants. Indeed the log report is

Complete output (5 lines):
  <mailto:git@github.com|git@github.com>: Permission denied (publickey).
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.
  ----------------------------------------

...so that seems to be the problem, but I'm flummoxed. I apologize if this is an obvious question; I looked under "Installing from version control or local files" to get the basic format, but searching for "private repository", "ssh", "publickey" etc didn't help on the docs site. What am I missing?

Hello! Check out https://www.pantsbuild.org/docs/proxies. I think you want "Using a custom cheeseshop/PyPI"

I don't think so, though--that seems to be for a custom PyPI-style repository (for packages that have been built and uploaded to a repo; as that says, "If you host your own wheels at a custom index"), not for installing a package from a source repository. That is covered in https://www.pantsbuild.org/docs/python-third-party-dependencies#using-custom-repositories, "Installing from Version Control and Local Files"... only putting the requirement into that format doesn't solve the SSH authentication problem.

I see. Does authentication work when you use

pip install mytarget@ <git+ssh://git>@github.com:/myorg/myrepo.git@myhash

without Pants? It can be helpful to remove the variable of Pants, e.g. so that we see if it's because, for example, Pants strips most env vars

Yes. I can copy and past the command attempted by pants (replacing the

****

it prints in place of the username) and it works fine in my current shell, which has ssh-agent running and populated with my keys. If I understand it right, pants runs in a sort of chroot so it's clean and reproducible (as you say, stripping environment vars and everything), which it should do. One way I've had to deal with this in docker containers is by either passing the agent through (not sure if pants has such a capability) or by executing a script which rewrites

~/.ssh/config

, but that can go a little awry with github (and probably wouldn't work if pants is isolating itself correctly).

Okay, that's really interesting and yes I do imagine the environment stripping could be in play. I'm in a meeting, but if you have a chance, could you please run try with this option https://www.pantsbuild.org/docs/troubleshooting#debug-tip-inspect-the-chroot-with---no-process-execution-cleanup-local-dirs, and go to the chroot and play around with the

__run.sh

script? In particular, you may need to tweak the env vars. Don't worry about breaking that bash script, it's safe to mess around with

Thanks; I'll do that.

Hmm this is an interesting problem

__run.sh

sets only relevant the environment variables for the process, so it should be fairly isolated.

So first question is how it works outside of Pants. Is this ssh-agent?

but it is possible that the

ssh-agent

requires a terminal to do its work?

Possible. It should be ssh-agent (I'm running ssh-agent through keychain so that it's always on with new shells because I got tired of manually starting it, but perhaps I need to figure out how to start a clean shell for some of these tests and just use ssh-agent, but that has worked in the past)

@echoing-farmer-15630: no need to share it, but do you have any environment variables related to SSH? perhaps

SSH_AUTH_SOCK

?

One solution of course would be to convince this particular developer to put his library in the monorepo with everyone else's, but tech solutions are often easier than social solutions 😉 Hmm, let me check... aha! Good thinking, @witty-crayon-22786; there is indeed an

SSH_AUTH_SOCK

for the agent.

so… there is a setting to allow these variable(s) to be passed through. but unfortunately, it has some hardcoded limits that were recently removed: https://www.pantsbuild.org/docs/reference-subprocess-environment

Hmm, that does look relevant (and indeed complains about its limits)

would you mind documenting the setup and which version you’re on in a new ticket, and then we can validate the fix?

Sure. Is there a way for me to test the

main

branch easily?

if you don’t mind getting your hands about … 45mins - 1hr dirty?

2.3.0 as of today.

ok: let me find a nearby SHA for that.

You're a hero, @witty-crayon-22786 🙂 Working...

huzzah.

Groovy, will look forward to that. Make a ticket for the backport reminder (and maybe documentation update for the custom repositories section to note that forwarding that variable may work?)

@echoing-farmer-15630: ah, please do then. it will make good documentation

https://github.com/pantsbuild/pants/issues/11790 and thanks for your help!

will try to get another 2.3.0rc out today.

and maybe documentation update for the custom repositories section to note that forwarding that variable may work?

once we get this all resolved, if you have time to make the doc edit it would be super appreciated!

Will be glad to do so.

thanks!