When pex is fetching requirements does it use `pip install`

Question

When pex is fetching requirements does it use `pip install` or does it do manual downloads with the file urls listed in the lockfile

enough-analyst-54434 · Answer

It uses a single `pip download` to create a lockfile perform the resolve in the 1st place When later resolving from a lockfile on a new machine or with a cold cache it uses parallel `pip download no deps` for each non `file ` URL needed

enough-analyst-54434 · Answer

The use of `pip download` for the individual URLs was only added later to make auth consistent between resolve lock and later retrieval The initial cut used urllib directly

high-yak-85899 · Answer

Thanks for the info I had tried playing around with proxying our dependencies through a private repository That repository caches dependencies for us whenever someone with proper permissions runs a `pip install` I would assume it also caches if one does `pip download` but I m not sure

enough-analyst-54434 · Answer

Yeah Pex never does an internet connected `pip install` It uses `pip download` to resolve `pip wheel` to turn any resolved sdists into wheels and then `pip install no deps` on those wheels individually in isolated chroots each to form PYTHONPATH elements that can be composed later when a PEX file boots up

happy-kitchen-89482 · Answer

`pip install` is just sending HTTP requests to the repository though right Unless the proxy is interpreting the request headers or some other shenaningans why would `pip install` do stuff that other HTTP requests for the same URLs wouldn t

enough-analyst-54434 · Answer

Well because of pip conf Pip might be sending a bunch of custom headers Pex or other fetchers would not be clued in on Pex in particular runs `pip isolated ` in all cases so can t see that config if present

enough-analyst-54434 · Answer

That would be a sane difference Agreed that if some proxy specifically does different things for a `pip` request maybe by user agent snooping that would be a bit crazy

enough-analyst-54434 · Answer

But Pex uses Pip here unfettered so the UA is Pips so the only difference I can see is the pip conf isolation

happy-kitchen-89482 · Answer

Yeah we agree that the only way to do pip specific stuff is to inspect the headers and it seems that this proxy is doing so I m just confused as to why it would be implemented thus

enough-analyst-54434 · Answer

Well but `pip download` sends no different headers from `pip install` This is why the only thing I can think of is removal of `pip conf` which Pex does is the difference here That removes auth headers of some sort is my guess

high-yak-85899 · Answer

I need to do some more experimentation on what works and all I would hope pip download would work the same Just all their docs demonstrate using pip install since that s what folks commonly use

enough-analyst-54434 · Answer

< high yak 85899> I really do think there is no way to detect from the server side the difference between `install`and `download`and I encourage you to look at security considerations `pip conf` if you haven t already That sort of thing will require different plumbing than `pip conf`to work with Pex through pip as things stand today

high-yak-85899 · Answer

Oh yeah I m not interested in treating them differently I just want to make sure the tool we are using isn t doing something fishy When I plumbed up credentials and a private registry in pants configuration things didn t work out of the box So I need to chase down a combo of me potentially doing something wrong and the expectations of the proxy service

enough-analyst-54434 · Answer

gt I just want to make sure the tool we are using isn t doing something fishy Yeah my weak claim is it can t be A tcpdump will not reveal the difference between an `install` and a `download`

high-yak-85899 · Answer

Is lock file generation done with pip download or are package needs gathered by some other mechanism

enough-analyst-54434 · Answer

Lock generation uses `pip download`

enough-analyst-54434 · Answer

and this may be getting there a PyPI API in certain cases Just a sec

enough-analyst-54434 · Answer

PEP 691 Code here Basically if the `pip download` logs indicate PEP 691 was used by Pip to fetch metadata then Pex must do the same in a post processing step to get sdist wheel hashes without downloading them all and hashing them Before PEP 691 with older Pip the hashes were listed in the `pip download` log as adornments to the download URLs Now though Pip no longer logs the fetched data thus the need to post process call the API ourselves So I expect those calls are missing auth info in some way or your service doesn t like the Pex UA which is `pex 2 1 133` for example here

enough-analyst-54434 · Answer

But to be clear < high yak 85899> this is only the case for creating a lock file and nowhere else in Pex code Just using a lock file won t trigger this PEP 691 API use since hashes are all known and written down in the lock file being read

high-yak-85899 · Answer

`pip download` does work with the proxy service as you expected

high-yak-85899 · Answer

Figured out that I was doing env var expansion wrong

high-yak-85899 · Answer

I was going `$ENV VAR` like the example in <https www pantsbuild org docs python third party dependencies authenticating to custom repos> but needed to be doing `$ env ENV VAR s`

high-yak-85899 · Answer

A very odd thing though Almost completed and then I got this error ```ERROR THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE If you have updated the package versions please update the hashes Otherwise examine the package contents carefully someone may have tampered with them apache flink libraries lt 1 16 2 gt =1 16 1 from <https artifactory corp astranis space artifactory api pypi general pypi remote packages packages 13 e4 0d9082016be1983ff28bebb771f0c6859ddcffb850015b37f758fc2eea1e apache flink libraries 1 16 1 tar gz sha256=f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11> from apache flink lt 1 17 gt =1 16 1 Expected sha256 f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11 Got 1906005054a583fd8685cb26496d3fce019632bdcae8b7022a31c4ab8dcca2f7``` But when I check what s cached in my proxy I see `f35f5bf0fe903c2e2f9f9617a5b8f4a9ed98ed5f12ad6d00b1e2341dfea55c11` correctly as the checksum

high-yak-85899 · Answer

oddly enough a second run worked fine

happy-kitchen-89482 · Answer

cosmic rays wink