https://pantsbuild.org/ logo
Join Slack
Powered by
Hey, does anyone know how to set up Pants to pull ...
# general
a
Hey, does anyone know how to set up Pants to pull Python distros from GCP's artifact registry? It seems like it can't find our package within our repository but I'm guessing it's actually an auth issue. The docs don't have much details on this. https://www.pantsbuild.org/2.23/docs/python/overview/third-party-dependencies#custom-repositories
Okay so I figured out the issue which is that the GCP registry requires you to have 
keyrings.google-artifactregistry-auth
installed. Pex provides a 
--keyring-provider
option so I just need a way to propagate that.
Okay looks like you still need to get 
keyrings.google-artifactregistry-aut
installed on the `PEX_PYTHON`path for this to work. I don't think there's a way to do that currently?
c
Yeah the bootstrapping-the-bootstrapper issue is thorny and I don't have a full example to point you at. My hand waving idea is that you could have the wheel 
keyrings.google-artifactregistry-auth
on a find links server/nfs-mount/checked-into-your-repo
a
I actually was trying to build a pex with pex itself as the entry point with the keyring installed but it just hangs at 
Downloading: file:///Users/nicholas.dellosa/Projects/my-repo/pex-cli.pex
And looks like the pex itself doesn't work either. But that should work in theory I would think?
@fast-nail-55400 Seems like you were able to get this to work?
f
Pex 2.29+ has support for a 
--keyring-provider
option to pass through to Pip.
It might be usable with Pants if 
--pex-cli-global-args
is set to include 
--keyring-provider=subprocess
and the 
keyring
binary is installed somewhere on the PATH seen by Pex.
Open question: what to do with Pex invocations that are not pex-cli.
@fast-nail-55400 Seems like you were able to get this to work?
Which is to say, that it did appear to work against AWS CodeArtifact in a proof of concept setting. Relevant Pants PRs are https://github.com/pantsbuild/pants/pull/21852 and https://github.com/pantsbuild/pants/pull/21853. Note: I'm not in a position any more to finish those.
So the use of 
--pex-cli-global-args
sketched above is likely the way forward for now.
a
So I tried to put 
keyring
and 
keyrings.google-artifactregistry-auth
in a pex together which I named 
keyring
and put in my 
/usr/local/bin
directory. Then I set 
global_args = ["--keyring-provider=subprocess"]
. Still not working, even though when I run 
keyring --list-backends
it looks good.
f
and is the 
PATH
for the pex-cli invocations including the installation location?
a
Yeah 
/usr/local/bin
is in there.
f
and is your find_links or index_url including https://USER@SITE/.... , I think the user part if necessary to force keyring usage. Plus pip_version needs to a recent Pip.
a
I'm wondering if there's some Python interpreter weirdness going on and keyring is silently failing. I set the shebang for the keyring pex to just 
python3
.
f
I found Pip is very finicky about when it uses keyring.
a
There's an option somewhere to get pex to spit out pip logs I think but I can't find it
f
You'd want to start with 
--pex-verbosity=9
and preserve sandboxes (
--keep-sandboxes=always
) and try to find where the 
pip.log
was written
What version of Python are you using?
a
hmm, pip.log is empty
3.8
f
There is a bootstrap problem (described in the Pex PR for 
--keyring-provider
) if only the vendored Pip in Pex (which does not support 
--keyring-provider
) is available.
Pex will warn about this and then proceed to invoke Pip without the 
--keyring-provider
option.
Python 3.12+ comes with a new enough Pip in its 
ensurepip
package.
The proof of concept was held together with lots of magic.
a
Copy code
2025-02-06T16:46:49,693 Keyring provider requested: subprocess
2025-02-06T16:46:49,693 Keyring provider set: subprocess with executable /usr/local/bin/keyring
2025-02-06T16:46:49,698 Status code 401 not in (200, 203, 300, 301, 308)
So it looks like it is using keyring but it's still getting unauthorized
f
Dumb question: Has the correct credential been written to keyring?
a
So based on the docs here (and I've successfully done this with plain pip) you just have to log in with the gcloud CLI and it should know where to look from there. I've got 
gcloud
on my path and am logged in, but maybe the pex can't see that?
f
Maybe? What behavior do you see if you make use of the preserved sandbox for the pex invocation?
For example, maybe an environment variable is missing from that invocation?
a
It's definitely that the credential is not set up correctly but I'm struggling to figure out why. When I install 
keyrings.google-artifactregistry-auth
into a virtual environment it all works like magic. But when I put it in a pex the credentials are not set up right.
Outside of Pants, if you just run a plain pex it should inherit all of the environment variables, minus the Python ones it does its thing with, right? I thought the issue is the home directory is not supplied by Pants env isolation but I hit the same issue with a plain 
pip install
so it's an issue with the keyring pex and not Pants.
f
Maybe avoid pex for now and create a regular venv with the correct packages installed and include its bin directory in PATH?
Just to side step pex issues for now.
a
Good idea, I'll try that out tomorrow.
Thanks for taking the time to help out!
👍 1
Good news is I think the problem is actually with the Google keyring for artifact registry. Even with plain pip it seems to only actually work with 
--keyring-provider=import
. When I set it to subprocess it fails the same way it does with pex.
f
The suggestion there is to make the username be 
oauth2accesstoken
a
Seems like they've removed that bit. 
keyring get <my registry url> ''
does work however.
I'm digging into the pip code now and my suspicion is that whatever it comes up with as the username is something different.
This is probably a very complicated question but is my understanding correct that the pip within pex is isolated from other stuff within the pex and that's why 
--keyring-provider=import
does not work with pex?
Copy code
Keyring provider requested: subprocess
Keyring provider set: subprocess with executable /usr/bin/keyring
/root/.cache/pex/installed_wheels/0/e7dae6694313f434a2727bf2906f27ad259bae090d7aa896590d86feec3d9d4a/google_auth-2.38.0-py2.py3-none-any.whl/google/auth/_default.py:76: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. See the following page for troubleshooting: <https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds>. 
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
Found credentials in keyring for us-central1-python.pkg.dev
Status code 401 not in (200, 203, 300, 301, 308)
Looking up "<my repo>/<my package>" in the cache
Request header has "max_age" as 0, cache bypassed
No cache entry available
<https://us-central1-python.pkg.dev:443> "GET <my project>/<my repo>/<my package> HTTP/1.1" 401 60
WARNING: 401 Error, Credentials not correct for <my repo>/<my package>
💀
It literally worked but then still didn't work
f
A few ideas: 1. Add a dummy 
keyring
trampoline script to log the exact env and args passed to it, and then exec the real 
keyring
binary (and tee stderr to a log). That might provide some insight into what is happening. 2. Inject a MitM proxy of some sort to inspect the actual request being sent.
a
I had a typo 😳
oath2accesstoken
instead of 
oauth2accesstoken
😂 1
f
Would you be willing to write up a recipe for how to configure everything for the Pants docs?
a
Yeah, it would honestly be nice to have Pants manage the keyring pex itself too. I can write up a feature request and submit a PR.
I'm seeing this issue in CI but not locally. Any idea how to make pex use a later pip version? I have 
--pip-version=latest
set but no dice.
Copy code
The --keyring-provider option is set to `subprocess`, but Pip v20.3.4+patched does not support the
`--keyring-provider` option (which is only available in Pip v23.1 and later versions).
Consequently, Pex is ignoring the --keyring-provider option for this particular Pip invocation.
Note: If this Pex invocation fails, it may be because Pex is trying to use its vendored Pip v20.3.4+patched
to bootstrap a newer Pip version which does support `--keyring-provider`, but you configured Pex/Pip
to use a Python package index which is not available without additional authentication.
In that case, you might wish to consider manually creating a `find-links` directory with that newer version
of Pip, so that Pex will still be able to install the newer version of Pip from the `find-links` directory
(which does not require authentication).
The invocation is still configured to use PyPi, so not sure why it would need a special find links directory.
Okay actually I was able to get the warning locally but it didn't fail.
ah wait I figured out it's actually failing for a different reason
f
that warning is probably from Pex using the vendored Pip (which does not support 
--keyring-provider
) to bootstrap a newer version of Pip which does support the option
the warning itself suggests a potential solution if applicable
(manually download the wheel for a newer Pip and store it in a directory in your repository and point 
find-links
at that directory)
I presume you have regular PyPi not available in this configuration?
a
No it is available, which is why I was confused. But the actual issue is that our CI on google cloud build is apparently not automatically logged into GCP.
👍 1
n
@ripe-gigabyte-88964 Did you get this working and could you share an example?
Hmm I seem to have something working with 
pants publish
, by adding a custom resolve for twine that contains the following requirements
Copy code
twine
keyring
keyrings.google-artifactregistry-auth
That's without doing anything with 
--keyring-provider
a
I got it working by using 
--keyring-provider=subprocess
and creating a custom 
keyring
binary that's actually a pex with 
keyring
and 
keyrings.google-artifactregistry-auth
provided and adding that to my 
PATH
.
✅ 1
👍 1
Then the index I added is 
"<https://oauth2accesstoken@YOUR_GAR_REGISTRY>"
n
I added the following in pants.toml - seems to work so far - will try to write it up somewhere once I've done more testing
Copy code
[python.resolves]
python-default = "3rdparty/python/default.lock"
twine = "3rdparty/python/twine.lock"

[twine]
install_from_resolve = "twine"
requirements = ["//:twine-reqs"]
Where :twine-reqs is the requirements file with reqs mentioned above. For registry pull (which I think this thread was originally about), I'm still using the oauth2token approach.
a
Want me to contribute something to the docs for this? Maybe something in here https://www.pantsbuild.org/2.24/docs/python/overview/third-party-dependencies
👍 1
s
It would be great to have a clear example, so far struggling to get this to work
a
Basically you just need to build a pex with 
keyring
and your provider in it, like this
Copy code
pex_binary(
    name="keyring",
    dependencies=[":tool_requirements#keyring", ":tool_requirements#keyrings.google-artifactregistry-auth"],
    resolve="tools",
    interpreter_constraints=[">=3.8,<=3.12"],
    sh_boot=True,
    entry_point="keyring",
    output_path="keyring",
)
and pop that pex onto your path somewhere, though not at 
/usr/local/bin
which pip is weird about for some reason. Then add your private registry to 
[python-repos].indexes
and under 
[pex-cli]
add 
global_args = ["--keyring-provider=subprocess"]
.
✅ 1
23 Views
Open in Slack
PreviousNext
Powered by