And one hopefully last one… Do we expect/desire any resolve caching for pex outside of pants?

I don't understand. Pex itself does resolve caching. Do you mean do we expect another PEX API user outside Pants to want to cache or want to modify caching?

(Caveat: I’m profiling two different things - making a pex file and running it, and just running with pex without making an intermediate pexfile) If I repeatedly run:

sudo py-spy record -o profile.svg -- python -m pex -D sample -m sample protobuf==3.11.3

to make a hello world pex which happens to have

protobuf

on its path, each invocation takes about 5 seconds, and about 17% of the time is spent in

resolve_multi

and about 20% in

add_distribution

Iff protobuf itself has no requirements or pinned requirements (or at least no unbounded requirements like >=3), we should expect performant caching, but we're relying on Pip and it may not have this. We could add short-circuit logic to skip resolves for fully pinned top-level requirements like the Pants plugin resolver code does.

How would you feel about an “--offline” flag, a la cargo, which says “Even in the face of unpinned deps, I’ll try to use what I have locally if I can”?

That seems reasonable. To avoid pex need to learn how to resolve again, I think this would just mean telling it ~

--no-index --find-links [cache]

The latter is not currently easy since the wheels cache is not layed out flat, but a symlink farm could be maintained.

Cool, I will have a play 🙂 Thanks!

Yeah, pip is slow here:

$ time pip download pex==2.1.6
Collecting pex==2.1.6
  Using cached pex-2.1.6-py2.py3-none-any.whl (2.3 MB)
  Saved ./pex-2.1.6-py2.py3-none-any.whl
Successfully downloaded pex

real	0m0.764s
user	0m0.532s
sys	0m0.084s

So, 3/4 a second to no-op on the pex distribution getting it from cache. Pex has 0 deps. Revealing is:

$ time pip -v download pex==2.1.6
Created temporary directory: /tmp/pip-req-tracker-4gk0y8g2
Initialized build tracking at /tmp/pip-req-tracker-4gk0y8g2
Created build tracker: /tmp/pip-req-tracker-4gk0y8g2
Entered build tracker: /tmp/pip-req-tracker-4gk0y8g2
Created temporary directory: /tmp/pip-download-2owo1nyb
1 location(s) to search for versions of pex:
* <https://pypi.org/simple/pex/>
Fetching project page and analyzing links: <https://pypi.org/simple/pex/>
Getting page <https://pypi.org/simple/pex/>
Found index url <https://pypi.org/simple>
Looking up "<https://pypi.org/simple/pex/>" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): <http://pypi.org:443|pypi.org:443>
<https://pypi.org:443> "GET /simple/pex/ HTTP/1.1" 304 0
  Found link <https://files.pythonhosted.org/packages/71/c2/5dbd8c218267bf8751ae42b83a47ae77c7372ae5127781b1975931e8b1e4/pex-0.1.0.tar.gz#sha256=d1c4818614c9ce375ac014f6bffca728487896a93f21ca4173d95fa621424f9c> (from <https://pypi.org/simple/pex/>), version: 0.1.0
...

So pip does the right thing per HTTP caches and expiry and all that. There could be a more appropriate distribution up there since there is no cache directive. By wheel rules, a pex wheel with a more specific set of tags would win if it suddenly appeared. This is all to say, an

--offline

option is probably good. However, it only buys me 200ms in this example:

$ time pip --cache-dir=/tmp/pip --isolated --disable-pip-version-check --no-python-version-warning download --dest /tmp/wheels --no-index -f /tmp/wheels pex==2.1.6 
Looking in links: /tmp/wheels
Processing /tmp/wheels/pex-2.1.6-py2.py3-none-any.whl
  File was already downloaded /tmp/wheels/pex-2.1.6-py2.py3-none-any.whl
Successfully downloaded pex

real	0m0.479s
user	0m0.408s
sys	0m0.071s