https://pantsbuild.org/ logo
#general
Title
# general
r

rich-london-74860

08/15/2022, 3:58 PM
Hello 👋, new to pants in the last week so this may be a stupid question. This is a specific question about deploying python packages, but also a general question about how things should be done. I need to deploy a python package to a private repository and the credentials for that private repository are stored in environment variables (For the sake of this question, please assume how these credentials are stored may not change). In general, it seems to me that with the exception of alterations to
pants.toml
, pants is not really designed to utilize environment variables, everything should be in a file, whether it’s check-in into the repo or a configuration file on the machine. I was able to workaround this in 2 ways. 1. One way I have this working is to use
./pants package
to produce the
*.whl
files and then use a
twine upload
(which uses environment variables to set the pypi target -
TWINE_REPOSITORY_URL
,
TWINE_USERNAME
,
TWINE_PASSWORD
) to publish those files (For the sake of this question, please assume that only python packages need to be published, meaning this would fulfill all of our needs). However, this does not use
./pants publish
and does not use the
repositories
property of the
python_distribution
target, which seems wrong to me. 2. Another way I had this working was to write a script that writes
~/.pypirc
based on credentials in environment variables. That script is built into a
pex_binary
target and run before
./pants publish
. When
./pants publish
runs, it picks up
~/.pypirc
. However, it seems like
pants
is designed to have all files either checked-in, a persisted fixture of the environment (the CI/CD worker instance has a
~/.pypirc
file saved on disk), or the product of a target specified in a
BUILD
file. In this case
~/.pypirc
is not any of these, so this feels very hacky to me.
h

high-yak-85899

08/15/2022, 4:00 PM
https://github.com/pantsbuild/pants/issues/16328 is really relevant to this. I think it was focused on pulling from a private repository, but it should be similar in a lot of ways.
e

enough-analyst-54434

08/15/2022, 4:54 PM
@rich-london-74860 you can interpolate env vars in
pants.toml
with
%(env.ENV_VAR_NAME)s
See here: https://www.pantsbuild.org/docs/options#config-file-interpolation
r

rich-london-74860

08/15/2022, 5:26 PM
@enough-analyst-54434 That seems like the right route for me, suppose I only have the URL for the pypi repository in environment variables, I do not know what it is at design time. Is it possible to interpolate environment variables in
BUILD
files? What do I put in the
repositories
parameter for
python_distribution
? In alternative 2 that I described above, I can put
"@alias"
and then use
[alias]
in the generated
.pypirc
file.
e

enough-analyst-54434

08/15/2022, 5:59 PM
I am not familiar with the
repositories
parameter for
python_distribution
.
But, reading your original post in full, I would classify an environment variable as exactly as hacky as ~/.pypirc - its local configuration not checked into the repo - and, presumably, that's with good intention!
The only hacky thing here is having to double-specify the local configuration. Presumably canonically in ~/.pypirc and then duplicated into an environment variable. It would be nice not to have to duplicate.
r

rich-london-74860

08/15/2022, 7:05 PM
That suggests to me then that option 1 is probably the simplest and best choice in this case.
e

enough-analyst-54434

08/15/2022, 7:06 PM
It may be the most expedient for you, yes. It would be great looking down the road though for the obvious thing, using
./pants publish
, to work easily.
r

rich-london-74860

08/22/2022, 7:25 PM
Following-up on this thread since I’m experiencing another issue that’s also related to the fact that we use a private pypi repository, but this time it’s regarding downloading packages. When running locally, pants correctly uses the configured additional indexes to install dependencies
Copy code
[python-repos]
indexes.add = ["%(env.PIP_EXTRA_INDEX_URL)s"]
which includes successfully authenticating with our private pypi repository (
PIP_EXTRA_INDEX_URL
includes basic-auth credentials) When running locally, but inside a docker container, which is using the same image as the one we use in CI/CD, the outcome is the same (so long as I pass
-e PIP_EXTRA_INDEX_URL=…
correctly) In our CI/CD system,
pip install my-package
is successful. The value set for
PIP_EXTRA_INDEX_URL
is the same and is correctly used by
pip
. However, in our CI/CD system, pants attempts to install dependencies form
PIP_EXTRA_INDEX_URL
, but fails to authenticate. It gets an error like:
Copy code
pex.result.ResultError: There were 2 errors downloading required artifacts:
1. my-package 0.1 from <https://my-private.repository.foo/pypi/pypi-local/my-package/0.1.0/my_package-0.1.0-py3-none-any.whl>
    HTTP Error 401: 
2. (another error just like the first ...)
Why would this fail to authenticate in one case, but not the other given that the value for
PIP_EXTRA_INDEX_URL
should include the correct credentials in all cases? Note that the URL in
PIP_EXTRA_INDEX_URL
is not in a private network, it is a publicly available url. Based on another problem I previously experienced, could this be because a file was ignored from git? I can’t find any other files in this repo that seem to be ignored, so which one might it be?
e

enough-analyst-54434

08/22/2022, 7:29 PM
To confirm, you're saying CI/C definitely has
PIP_EXTRA_INDEX_URL
exported with the expected correct value and visible to Pants? What version of Pants is this?
r

rich-london-74860

08/22/2022, 7:30 PM
you’re saying CI/C definitely has
PIP_EXTRA_INDEX_URL
exported with the expected correct value
Yes
and visible to Pants?
Is it possible that
pants
modifies the value of
PIP_EXTRA_INDEX_URL
? Is it possible to make
pants
print out what it thinks this value is?
What version of Pants is this?
2.12.0
Are you willing to try Pants 2.13.0rc1? That has Pex 2.1.102: https://pypi.org/project/pantsbuild.pants/2.13.0rc1/
r

rich-london-74860

08/22/2022, 7:36 PM
I’ll give it a shot
Same error, but with more content in the error message.
e

enough-analyst-54434

08/22/2022, 8:01 PM
Can you provide the full command line + output?
r

rich-london-74860

08/22/2022, 8:10 PM
I can with some names obfuscated
I have replaced urls and names in this snippet so I hope it’s still helpful:
Copy code
There were 2 errors downloading required artifacts:
1. package1 0.9 from <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl>
    ERROR: Could not install requirement package1==0.1.0 from <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl> because of HTTP error 401 Client Error:  for url: <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl> for URL <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl>
2. package2 0.21 from <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl>
    ERROR: Could not install requirement package2==0.2.0 from <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl> because of HTTP error 401 Client Error:  for url: <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl> for URL <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl>
I noticed that the basic authentication credentials have been stripped from these logs, which I assume is intentional.
e

enough-analyst-54434

08/22/2022, 8:16 PM
Yes, that's intentional. Do you have a basic sanity check yet in CI where you print out the env var just before executing
./pants ...
?
r

rich-london-74860

08/22/2022, 8:17 PM
Yes, I effectively do
pip install package1 package2
and it works
I’m also dumping all env vars and they are correct
e

enough-analyst-54434

08/22/2022, 8:18 PM
Ok. Thinking....
r

rich-london-74860

08/22/2022, 8:21 PM
Is it because the basic authentication credentials are included in the URL? Is there another way to configure that?
e

enough-analyst-54434

08/22/2022, 8:21 PM
Pex works with either creds in the URL or via ~/.netrc. Either works, both are pretty well tested.
Ok, so the exmaple file URL is https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl what is the extra index with the basic auth info removed from the URL?
Is it https://private-repo.repo.com as well? KIf so what is the full URL including any extra path components?
Another tactic here - can you reveal the implementation of your private repo? For example, is it JFrog? If its some product I can try to set up and test against, that might be fastest for debugging, In https://github.com/pantsbuild/pex/issues/1803 it was AWS Code Artifact being used by the person who reported the issue; so I just set one up myself to confirm the bug and then confirm the fix.
r

rich-london-74860

08/22/2022, 8:52 PM
Sorry, it’s not
<http://private-repo.repo.com|private-repo.repo.com>
, I just made that up to hide some private information in the actual url. But also, I figured it out. Since I need to maintain multiple sets of 3rd-party dependencies, I’m using
resolves
which requires generating lock files locally. If the url in the lock file does not match the url used for
PIP_EXTRA_INDEX_URL
then the same basic authentication credentials will not be used (which makes sense). We have multiple sub-domains that redirect to the same server (something like
<http://private-repo.repo.com|private-repo.repo.com>
vs
<http://my-private-repo.repo.com|my-private-repo.repo.com>
), I had a different sub-domain configured locally than what’s configured in CI/CD.
e

enough-analyst-54434

08/22/2022, 8:53 PM
Aha, that's the problem then I think: https://github.com/pantsbuild/pex/blob/3e71bc7896935a8177e9bbe9399769c3522528dc/pex/auth.py#L44-L48 Right now Pex reduces the URL to the host-port of the index and only injects the auth for matching hosts.
Perhaps this should be reduced further (really expanded), to injecting the auth for the whole domain. So if the index is https://pypi.org/simple then the auth still gets injected for https://file.pypi.org
@rich-london-74860 do you agree that sounds like the problem here? I.E: Your index has a different hostname portion or its URL than where the files are served from?
👍 1
Or are you saying this was all just a misconfiguration on your end and you're all set now with a solution?
r

rich-london-74860

08/22/2022, 8:57 PM
So if the index is https://pypi.org/simple then the auth still gets injected for https://file.pypi.org
Yes, this would have fixed my problem, but I wonder if using the same credentials for different sub-domains is universally desirable?
e

enough-analyst-54434

08/22/2022, 8:59 PM
It definitely might not be! That's why host / port was the furthest I reduced thi in the 1st place. Even then that felt risky. Folks can have different auth per path within a host if that's how they partition their applications for example.
Ok, I'll leave this problem for another day then since it sounds like you have a working solution now @rich-london-74860
👍 1
5 Views