high-yak-8589904/18/2023, 11:13 PM
or does it do manual downloads with the file urls listed in the lockfile?
enough-analyst-5443404/18/2023, 11:19 PM
to create a lockfile (perform the resolve) in the 1st place. When later resolving from a lockfile on a new machine or with a cold cache it uses parallel
for each non-
pip download --no-deps
for the individual URLs was only added later to make auth consistent between resolve (lock) and later retrieval. The initial cut used urllib directly.
high-yak-8589904/18/2023, 11:29 PM
. I would assume it also caches if one does
, but I'm not sure.
enough-analyst-5443404/18/2023, 11:31 PM
. It uses
to turn any resolved sdists into wheels, and then
on those wheels individually in isolated chroots each to form PYTHONPATH elements that can be composed later when a PEX file boots up.
pip install --no-deps
happy-kitchen-8948204/19/2023, 1:07 AM
is just sending HTTP requests to the repository though, right? Unless the proxy is interpreting the request headers or some other shenaningans, why would
do stuff that other HTTP requests for the same URLs wouldn't?
enough-analyst-5443404/19/2023, 1:09 AM
in all cases; so can't see that config if present.
pip --isolated ...
request - maybe by user agent snooping? - that would be a bit crazy.
happy-kitchen-8948204/19/2023, 2:53 AM
enough-analyst-5443404/19/2023, 3:19 AM
sends no different headers from
. This is why the only thing I can think of is removal of
- which Pex does - is the difference here. That removes auth headers of some sort is my guess.
high-yak-8589904/19/2023, 2:48 PM
enough-analyst-5443404/19/2023, 3:13 PM
if you haven't already. That sort of thing will require different plumbing than `pip.conf`to work with Pex through pip as things stand today.
high-yak-8589904/19/2023, 3:19 PM
enough-analyst-5443404/19/2023, 3:19 PM
I just want to make sure the tool we are using isn’t doing something fishy.Yeah - my weak claim is it can't be. A tcpdump will not reveal the difference between an
high-yak-8589904/19/2023, 3:24 PM
enough-analyst-5443404/19/2023, 3:37 PM
logs indicate PEP-691 was used by Pip to fetch metadata, then Pex must do the same in a post-processing step to get sdist wheel hashes without downloading them all and hashing them: https://github.com/pantsbuild/pex/blob/93e904ade594654da4455fd61dacb968d56d5fd8/pex/resolve/locker.py#L409-L421 Before PEP-691 / with older Pip, the hashes were listed in the
log as adornments to the download URLs. Now, though, Pip no longer logs the fetched data; thus the need to post-process call the API ourselves. So I expect those calls are missing auth info in some way or your service doesn't like the Pex UA, which is
(for example) here: https://github.com/pantsbuild/pex/blob/8c14e563493b8f4a85b4a3c735596dd55586a52b/pex/fetcher.py#L32-L33
high-yak-8589904/19/2023, 9:02 PM
does work with the proxy service as you expected
But when I check what's cached in my proxy, I see
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. apache-flink-libraries<1.16.2,>=1.16.1 from <https://artifactory.corp.astranis.space/artifactory/api/pypi/general-pypi-remote/packages/packages/13/e4/0d9082016be1983ff28bebb771f0c6859ddcffb850015b37f758fc2eea1e/apache-flink-libraries-1.16.1.tar.gz#sha256=f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11> (from apache-flink<1.17,>=1.16.1): Expected sha256 f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11 Got 1906005054a583fd8685cb26496d3fce019632bdcae8b7022a31c4ab8dcca2f7
correctly as the checksum
happy-kitchen-8948204/19/2023, 10:04 PM