Hello 👋, new to pants in the last week so this may be a stupid question. This is a specific question about deploying python packages, but also a general question about how things should be done. I need to deploy a python package to a private repository and the credentials for that private repository are stored in environment variables (For the sake of this question, please assume how these credentials are stored may not change). In general, it seems to me that with the exception of alterations to

pants.toml

, pants is not really designed to utilize environment variables, everything should be in a file, whether it’s check-in into the repo or a configuration file on the machine. I was able to workaround this in 2 ways. 1. One way I have this working is to use

./pants package

to produce the

*.whl

files and then use a

twine upload

(which uses environment variables to set the pypi target -

TWINE_REPOSITORY_URL

,

TWINE_USERNAME

,

TWINE_PASSWORD

) to publish those files (For the sake of this question, please assume that only python packages need to be published, meaning this would fulfill all of our needs). However, this does not use

./pants publish

and does not use the

repositories

property of the

python_distribution

target, which seems wrong to me. 2. Another way I had this working was to write a script that writes

~/.pypirc

based on credentials in environment variables. That script is built into a

pex_binary

target and run before

./pants publish

. When

./pants publish

runs, it picks up

~/.pypirc

. However, it seems like

pants

is designed to have all files either checked-in, a persisted fixture of the environment (the CI/CD worker instance has a

~/.pypirc

file saved on disk), or the product of a target specified in a

BUILD

file. In this case

~/.pypirc

is not any of these, so this feels very hacky to me.

https://github.com/pantsbuild/pants/issues/16328 is really relevant to this. I think it was focused on pulling from a private repository, but it should be similar in a lot of ways.

@rich-london-74860 you can interpolate env vars in

pants.toml

with

%(env.ENV_VAR_NAME)s

See here: https://www.pantsbuild.org/docs/options#config-file-interpolation

@enough-analyst-54434 That seems like the right route for me, suppose I only have the URL for the pypi repository in environment variables, I do not know what it is at design time. Is it possible to interpolate environment variables in

BUILD

files? What do I put in the

repositories

parameter for

python_distribution

? In alternative 2 that I described above, I can put

"@alias"

and then use

[alias]

in the generated

.pypirc

file.

I am not familiar with the

repositories

parameter for

python_distribution

.

That suggests to me then that option 1 is probably the simplest and best choice in this case.

It may be the most expedient for you, yes. It would be great looking down the road though for the obvious thing, using

./pants publish

, to work easily.

Following-up on this thread since I’m experiencing another issue that’s also related to the fact that we use a private pypi repository, but this time it’s regarding downloading packages. When running locally, pants correctly uses the configured additional indexes to install dependencies

[python-repos]
indexes.add = ["%(env.PIP_EXTRA_INDEX_URL)s"]

which includes successfully authenticating with our private pypi repository (

PIP_EXTRA_INDEX_URL

includes basic-auth credentials) When running locally, but inside a docker container, which is using the same image as the one we use in CI/CD, the outcome is the same (so long as I pass

-e PIP_EXTRA_INDEX_URL=…

correctly) In our CI/CD system,

pip install my-package

is successful. The value set for

PIP_EXTRA_INDEX_URL

is the same and is correctly used by

pip

. However, in our CI/CD system, pants attempts to install dependencies form

PIP_EXTRA_INDEX_URL

, but fails to authenticate. It gets an error like:

pex.result.ResultError: There were 2 errors downloading required artifacts:
1. my-package 0.1 from <https://my-private.repository.foo/pypi/pypi-local/my-package/0.1.0/my_package-0.1.0-py3-none-any.whl>
    HTTP Error 401: 
2. (another error just like the first ...)

Why would this fail to authenticate in one case, but not the other given that the value for

PIP_EXTRA_INDEX_URL

should include the correct credentials in all cases? Note that the URL in

PIP_EXTRA_INDEX_URL

is not in a private network, it is a publicly available url. Based on another problem I previously experienced, could this be because a file was ignored from git? I can’t find any other files in this repo that seem to be ignored, so which one might it be?

To confirm, you're saying CI/C definitely has

PIP_EXTRA_INDEX_URL

exported with the expected correct value and visible to Pants? What version of Pants is this?

you’re saying CI/C definitely has

PIP_EXTRA_INDEX_URL

exported with the expected correct value

Yes

and visible to Pants?

Is it possible that

pants

modifies the value of

PIP_EXTRA_INDEX_URL

? Is it possible to make

pants

print out what it thinks this value is?

What version of Pants is this?

2.12.0

2.12.0 is on Pex 2.1.90: https://www.pantsbuild.org/docs/reference-pex-cli#section-version and I suspect your fix is in Pex 2.1.93: https://github.com/pantsbuild/pex/releases/tag/v2.1.93 - namely https://github.com/pantsbuild/pex/pull/1811 which fixes https://github.com/pantsbuild/pex/issues/1803

I’ll give it a shot

Can you provide the full command line + output?

I can with some names obfuscated

Yes, that's intentional. Do you have a basic sanity check yet in CI where you print out the env var just before executing

./pants ...

?

Yes, I effectively do

pip install package1 package2

and it works

Ok. Thinking....

Is it because the basic authentication credentials are included in the URL? Is there another way to configure that?

Pex works with either creds in the URL or via ~/.netrc. Either works, both are pretty well tested.

Sorry, it’s not

<http://private-repo.repo.com|private-repo.repo.com>

, I just made that up to hide some private information in the actual url. But also, I figured it out. Since I need to maintain multiple sets of 3rd-party dependencies, I’m using

resolves

which requires generating lock files locally. If the url in the lock file does not match the url used for

PIP_EXTRA_INDEX_URL

then the same basic authentication credentials will not be used (which makes sense). We have multiple sub-domains that redirect to the same server (something like

<http://private-repo.repo.com|private-repo.repo.com>

vs

<http://my-private-repo.repo.com|my-private-repo.repo.com>

), I had a different sub-domain configured locally than what’s configured in CI/CD.

Aha, that's the problem then I think: https://github.com/pantsbuild/pex/blob/3e71bc7896935a8177e9bbe9399769c3522528dc/pex/auth.py#L44-L48 Right now Pex reduces the URL to the host-port of the index and only injects the auth for matching hosts.

So if the index is https://pypi.org/simple then the auth still gets injected for https://file.pypi.org

Yes, this would have fixed my problem, but I wonder if using the same credentials for different sub-domains is universally desirable?

It definitely might not be! That's why host / port was the furthest I reduced thi in the 1st place. Even then that felt risky. Folks can have different auth per path within a host if that's how they partition their applications for example.