Hi folks, while trying to update from pants 2.20, I suppose I hit a regression when generating a Python lockfile that includes a wheel in a private package index. We have a lockfile with a package from a private index, whose package URL includes credentials and looks like

package @ <https://secret-name>:@pypi.url.com/packages/package.whl#sha256=<hash>

(the credentials have an empty password). We don't specify

<http://pypi.url.com|pypi.url.com>

in

python-repos.indexes

of

pants.toml

because that would pull in other dependencies from that index, which we don't want. We have other indexes specified there for other packages. Up to (and including) pants 2.20 (pex 2.1.163), generating a lockfile worked as advertised, and the secret package URL makes it into the lockfile. Starting with pants 2.21 (pex 2.3.1), lockfile generation fails with this error during pex download:

pid [...] -> [...]/bin/python [...]/pex --disable-pip-version-check --exists-action a --no-input --no-python-version-warning --isolated --log [...]/.tmp/pex-pip-log.cvdej1rr/pip.log -q --cache-dir [...]/pip_cache download --dest [...]/fingerprint_artifact.ri_3r9h6 --no-deps package @ <https://pypi.url.com/packages/package.whl> --index-url [...] --retries 5 --timeout 15 exited with 1 and STDERR:

pip:   ERROR: HTTP error 401 while getting <https://pypi.url.com/packages/package.whl>

pip: ERROR: Could not install requirement package==[...] from <https://pypi.url.com/packages/package.whl> because of HTTP error 401 Client Error: Unauthorized for url: <https://pypi.url.com/packages/package.whl> for URL <https://pypi.url.com/packages/package.whl>

The crucial observation here is that the credentials are missing from the URL. I manually ran the command again with credentials in the package URL and it worked as expected. When inspecting sandbox created during lockfile generation, I enabled preservation of

pip.log

and observed that the

pex lock create

command succeeds. Afterwards, the lockfile generation seems to run this

pex download

command (without package URL credentials), which then fails. This observation leads me to suspect that pants may be involved here (rather than pex), which is why I'm posting here for advice. Thank you in advance!

Can you repro this on 2.22 ... 2.25? I have faint memories that this has been reported and potentially fixed, but it might be a different issue.

Yep, I've seen this issue in 2.21, 2.22, 2.23 and 2.25.

Ack, thanks! Can you file a GitHub issue (or bump an existing one)?

Given Pex controls lock file generation, this is probably a Pex issue. The Pants upgrades imply Pex upgrade.

This looks like it might be a side effect of https://github.com/pex-tool/pex/pull/2349

Thanks for the idea, highly appreciated! The keyring approach sounds interesting, wasn't aware of that. I also like the netrc idea (didn't know pex reads that!), but unfortunately that's hard-coded to use

~/.netrc

, requiring our developers to update that file each time our credentials expire. I think I'll set up (yet another) internal package index and use that one for the private packages, which are causing the issue. It's a bit more infra but the tooling configuration remains at existing level of complexity. Thanks again!

Well, if your creds expire periodically then you don’t want to embed them in your lockfile anyway…