high-yak-85899
04/18/2023, 11:13 PMpip install
or does it do manual downloads with the file urls listed in the lockfile?enough-analyst-54434
04/18/2023, 11:19 PMpip download
to create a lockfile (perform the resolve) in the 1st place. When later resolving from a lockfile on a new machine or with a cold cache it uses parallel pip download --no-deps
for each non-file://
URL needed.pip download
for the individual URLs was only added later to make auth consistent between resolve (lock) and later retrieval. The initial cut used urllib directly.high-yak-85899
04/18/2023, 11:29 PMpip install
. I would assume it also caches if one does pip download
, but I'm not sure.enough-analyst-54434
04/18/2023, 11:31 PMpip install
. It uses pip download
to resolve, pip wheel
to turn any resolved sdists into wheels, and then pip install --no-deps
on those wheels individually in isolated chroots each to form PYTHONPATH elements that can be composed later when a PEX file boots up.happy-kitchen-89482
04/19/2023, 1:07 AMpip install
is just sending HTTP requests to the repository though, right? Unless the proxy is interpreting the request headers or some other shenaningans, why would pip install
do stuff that other HTTP requests for the same URLs wouldn't?enough-analyst-54434
04/19/2023, 1:09 AMpip --isolated ...
in all cases; so can't see that config if present.pip
request - maybe by user agent snooping? - that would be a bit crazy.happy-kitchen-89482
04/19/2023, 2:53 AMenough-analyst-54434
04/19/2023, 3:19 AMpip download
sends no different headers from pip install
. This is why the only thing I can think of is removal of pip.conf
- which Pex does - is the difference here. That removes auth headers of some sort is my guess.high-yak-85899
04/19/2023, 2:48 PMenough-analyst-54434
04/19/2023, 3:13 PMpip.conf
if you haven't already. That sort of thing will require different plumbing than `pip.conf`to work with Pex through pip as things stand today.high-yak-85899
04/19/2023, 3:19 PMenough-analyst-54434
04/19/2023, 3:19 PMI just want to make sure the tool we are using isn’t doing something fishy.Yeah - my weak claim is it can't be. A tcpdump will not reveal the difference between an
install
and a download
.high-yak-85899
04/19/2023, 3:24 PMenough-analyst-54434
04/19/2023, 3:37 PMpip download
.pip download
logs indicate PEP-691 was used by Pip to fetch metadata, then Pex must do the same in a post-processing step to get sdist wheel hashes without downloading them all and hashing them: https://github.com/pantsbuild/pex/blob/93e904ade594654da4455fd61dacb968d56d5fd8/pex/resolve/locker.py#L409-L421
Before PEP-691 / with older Pip, the hashes were listed in the pip download
log as adornments to the download URLs. Now, though, Pip no longer logs the fetched data; thus the need to post-process call the API ourselves. So I expect those calls are missing auth info in some way or your service doesn't like the Pex UA, which is pex/2.1.133
(for example) here: https://github.com/pantsbuild/pex/blob/8c14e563493b8f4a85b4a3c735596dd55586a52b/pex/fetcher.py#L32-L33high-yak-85899
04/19/2023, 9:02 PMpip download
does work with the proxy service as you expected$ENV_VAR
like the example in https://www.pantsbuild.org/docs/python-third-party-dependencies#authenticating-to-custom-repos but needed to be doing $(env.ENV_VAR)s
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
apache-flink-libraries<1.16.2,>=1.16.1 from <https://artifactory.corp.astranis.space/artifactory/api/pypi/general-pypi-remote/packages/packages/13/e4/0d9082016be1983ff28bebb771f0c6859ddcffb850015b37f758fc2eea1e/apache-flink-libraries-1.16.1.tar.gz#sha256=f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11> (from apache-flink<1.17,>=1.16.1):
Expected sha256 f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11
Got 1906005054a583fd8685cb26496d3fce019632bdcae8b7022a31c4ab8dcca2f7
But when I check what's cached in my proxy, I see f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11
correctly as the checksumhappy-kitchen-89482
04/19/2023, 10:04 PM