https://pantsbuild.org/ logo
#pex
Title
# pex
a

average-vr-56795

03/23/2020, 2:35 PM
And one hopefully last one… Do we expect/desire any resolve caching for pex outside of pants?
e

enough-analyst-54434

03/23/2020, 2:41 PM
I don't understand. Pex itself does resolve caching. Do you mean do we expect another PEX API user outside Pants to want to cache or want to modify caching?
a

average-vr-56795

03/23/2020, 2:45 PM
(Caveat: I’m profiling two different things - making a pex file and running it, and just running with pex without making an intermediate pexfile) If I repeatedly run:
sudo py-spy record -o profile.svg -- python -m pex -D sample -m sample protobuf==3.11.3
to make a hello world pex which happens to have
protobuf
on its path, each invocation takes about 5 seconds, and about 17% of the time is spent in
resolve_multi
and about 20% in
add_distribution
Do we expect or want any caching so that we’re not spending about 1s in
resolve_multi
to look up a thing we already looked up?
e

enough-analyst-54434

03/23/2020, 3:03 PM
Iff protobuf itself has no requirements or pinned requirements (or at least no unbounded requirements like >=3), we should expect performant caching, but we're relying on Pip and it may not have this. We could add short-circuit logic to skip resolves for fully pinned top-level requirements like the Pants plugin resolver code does.
👍 1
The latter will require writing a Pip-compatible requirements.txt parser, but that will also benefit https://github.com/pantsbuild/pex/issues/899 which needs it.
a

average-vr-56795

03/23/2020, 3:07 PM
How would you feel about an “--offline” flag, a la cargo, which says “Even in the face of unpinned deps, I’ll try to use what I have locally if I can”?
e

enough-analyst-54434

03/23/2020, 3:27 PM
That seems reasonable. To avoid pex need to learn how to resolve again, I think this would just mean telling it ~
--no-index --find-links [cache]
The latter is not currently easy since the wheels cache is not layed out flat, but a symlink farm could be maintained.
👍 1
a

average-vr-56795

03/23/2020, 3:27 PM
Cool, I will have a play 🙂 Thanks!
e

enough-analyst-54434

03/23/2020, 9:53 PM
Yeah, pip is slow here:
Copy code
$ time pip download pex==2.1.6
Collecting pex==2.1.6
  Using cached pex-2.1.6-py2.py3-none-any.whl (2.3 MB)
  Saved ./pex-2.1.6-py2.py3-none-any.whl
Successfully downloaded pex

real	0m0.764s
user	0m0.532s
sys	0m0.084s
So, 3/4 a second to no-op on the pex distribution getting it from cache. Pex has 0 deps. Revealing is:
Copy code
$ time pip -v download pex==2.1.6
Created temporary directory: /tmp/pip-req-tracker-4gk0y8g2
Initialized build tracking at /tmp/pip-req-tracker-4gk0y8g2
Created build tracker: /tmp/pip-req-tracker-4gk0y8g2
Entered build tracker: /tmp/pip-req-tracker-4gk0y8g2
Created temporary directory: /tmp/pip-download-2owo1nyb
1 location(s) to search for versions of pex:
* <https://pypi.org/simple/pex/>
Fetching project page and analyzing links: <https://pypi.org/simple/pex/>
Getting page <https://pypi.org/simple/pex/>
Found index url <https://pypi.org/simple>
Looking up "<https://pypi.org/simple/pex/>" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): <http://pypi.org:443|pypi.org:443>
<https://pypi.org:443> "GET /simple/pex/ HTTP/1.1" 304 0
  Found link <https://files.pythonhosted.org/packages/71/c2/5dbd8c218267bf8751ae42b83a47ae77c7372ae5127781b1975931e8b1e4/pex-0.1.0.tar.gz#sha256=d1c4818614c9ce375ac014f6bffca728487896a93f21ca4173d95fa621424f9c> (from <https://pypi.org/simple/pex/>), version: 0.1.0
...
So pip does the right thing per HTTP caches and expiry and all that. There could be a more appropriate distribution up there since there is no cache directive. By wheel rules, a pex wheel with a more specific set of tags would win if it suddenly appeared. This is all to say, an
--offline
option is probably good. However, it only buys me 200ms in this example:
Copy code
$ time pip --cache-dir=/tmp/pip --isolated --disable-pip-version-check --no-python-version-warning download --dest /tmp/wheels --no-index -f /tmp/wheels pex==2.1.6 
Looking in links: /tmp/wheels
Processing /tmp/wheels/pex-2.1.6-py2.py3-none-any.whl
  File was already downloaded /tmp/wheels/pex-2.1.6-py2.py3-none-any.whl
Successfully downloaded pex

real	0m0.479s
user	0m0.408s
sys	0m0.071s