rich-london-74860
08/15/2022, 3:58 PMpants.toml
, pants is not really designed to utilize environment variables, everything should be in a file, whether it’s check-in into the repo or a configuration file on the machine. I was able to workaround this in 2 ways.
1. One way I have this working is to use ./pants package
to produce the *.whl
files and then use a twine upload
(which uses environment variables to set the pypi target - TWINE_REPOSITORY_URL
, TWINE_USERNAME
, TWINE_PASSWORD
) to publish those files (For the sake of this question, please assume that only python packages need to be published, meaning this would fulfill all of our needs). However, this does not use ./pants publish
and does not use the repositories
property of the python_distribution
target, which seems wrong to me.
2. Another way I had this working was to write a script that writes ~/.pypirc
based on credentials in environment variables. That script is built into a pex_binary
target and run before ./pants publish
. When ./pants publish
runs, it picks up ~/.pypirc
. However, it seems like pants
is designed to have all files either checked-in, a persisted fixture of the environment (the CI/CD worker instance has a ~/.pypirc
file saved on disk), or the product of a target specified in a BUILD
file. In this case ~/.pypirc
is not any of these, so this feels very hacky to me.high-yak-85899
08/15/2022, 4:00 PMenough-analyst-54434
08/15/2022, 4:54 PMpants.toml
with %(env.ENV_VAR_NAME)s
See here: https://www.pantsbuild.org/docs/options#config-file-interpolationrich-london-74860
08/15/2022, 5:26 PMBUILD
files? What do I put in the repositories
parameter for python_distribution
?
In alternative 2 that I described above, I can put "@alias"
and then use [alias]
in the generated .pypirc
file.enough-analyst-54434
08/15/2022, 5:59 PMrepositories
parameter for python_distribution
.rich-london-74860
08/15/2022, 7:05 PMenough-analyst-54434
08/15/2022, 7:06 PM./pants publish
, to work easily.rich-london-74860
08/22/2022, 7:25 PM[python-repos]
indexes.add = ["%(env.PIP_EXTRA_INDEX_URL)s"]
which includes successfully authenticating with our private pypi repository (PIP_EXTRA_INDEX_URL
includes basic-auth credentials)
When running locally, but inside a docker container, which is using the same image as the one we use in CI/CD, the outcome is the same (so long as I pass -e PIP_EXTRA_INDEX_URL=…
correctly)
In our CI/CD system, pip install my-package
is successful. The value set for PIP_EXTRA_INDEX_URL
is the same and is correctly used by pip
.
However, in our CI/CD system, pants attempts to install dependencies form PIP_EXTRA_INDEX_URL
, but fails to authenticate. It gets an error like:
pex.result.ResultError: There were 2 errors downloading required artifacts:
1. my-package 0.1 from <https://my-private.repository.foo/pypi/pypi-local/my-package/0.1.0/my_package-0.1.0-py3-none-any.whl>
HTTP Error 401:
2. (another error just like the first ...)
Why would this fail to authenticate in one case, but not the other given that the value for PIP_EXTRA_INDEX_URL
should include the correct credentials in all cases?
Note that the URL in PIP_EXTRA_INDEX_URL
is not in a private network, it is a publicly available url.
Based on another problem I previously experienced, could this be because a file was ignored from git? I can’t find any other files in this repo that seem to be ignored, so which one might it be?enough-analyst-54434
08/22/2022, 7:29 PMPIP_EXTRA_INDEX_URL
exported with the expected correct value and visible to Pants? What version of Pants is this?rich-london-74860
08/22/2022, 7:30 PMyou’re saying CI/C definitely hasYesexported with the expected correct valuePIP_EXTRA_INDEX_URL
and visible to Pants?Is it possible that
pants
modifies the value of PIP_EXTRA_INDEX_URL
? Is it possible to make pants
print out what it thinks this value is?
What version of Pants is this?2.12.0
enough-analyst-54434
08/22/2022, 7:34 PMrich-london-74860
08/22/2022, 7:36 PMenough-analyst-54434
08/22/2022, 8:01 PMrich-london-74860
08/22/2022, 8:10 PMThere were 2 errors downloading required artifacts:
1. package1 0.9 from <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl>
ERROR: Could not install requirement package1==0.1.0 from <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl> because of HTTP error 401 Client Error: for url: <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl> for URL <https://private-repo.repo.com/api/pypi/pypi-local/package1/0.1.0/package1_sdk-0.1.0-py3-none-any.whl>
2. package2 0.21 from <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl>
ERROR: Could not install requirement package2==0.2.0 from <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl> because of HTTP error 401 Client Error: for url: <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl> for URL <https://private-repo.repo.com/api/pypi/pypi-local/package2/0.2.0/package2-0.2.0-py3-none-any.whl>
I noticed that the basic authentication credentials have been stripped from these logs, which I assume is intentional.enough-analyst-54434
08/22/2022, 8:16 PM./pants ...
?rich-london-74860
08/22/2022, 8:17 PMpip install package1 package2
and it worksenough-analyst-54434
08/22/2022, 8:18 PMrich-london-74860
08/22/2022, 8:21 PMenough-analyst-54434
08/22/2022, 8:21 PMrich-london-74860
08/22/2022, 8:52 PM<http://private-repo.repo.com|private-repo.repo.com>
, I just made that up to hide some private information in the actual url.
But also, I figured it out. Since I need to maintain multiple sets of 3rd-party dependencies, I’m using resolves
which requires generating lock files locally. If the url in the lock file does not match the url used for PIP_EXTRA_INDEX_URL
then the same basic authentication credentials will not be used (which makes sense). We have multiple sub-domains that redirect to the same server (something like <http://private-repo.repo.com|private-repo.repo.com>
vs <http://my-private-repo.repo.com|my-private-repo.repo.com>
), I had a different sub-domain configured locally than what’s configured in CI/CD.enough-analyst-54434
08/22/2022, 8:53 PMrich-london-74860
08/22/2022, 8:57 PMSo if the index is https://pypi.org/simple then the auth still gets injected for https://file.pypi.orgYes, this would have fixed my problem, but I wonder if using the same credentials for different sub-domains is universally desirable?
enough-analyst-54434
08/22/2022, 8:59 PM