https://pantsbuild.org/ logo
Join Slack
Powered by
I have a question about pex venvs and hard links. ...
# general
a
I have a question about pex venvs and hard links. We unpack the same large pex file into multiple venvs on the same machine, and I want to make sure I'm doing everything possible to make the venv unpacking is fast and efficient. Do I understand correctly that the pex file itself is unpacked into PEX_ROOT once and the same package files are then shared via hard links into every venv that is created from that pex file? Is there anything I can do to make the second 
pex-tools my-pex.pex venv /here
faster?
e
You understand correctly, but I also encourage you to check using 
stat
! There are no low-hanging fruit to speed up venv creation that I'm aware of, but it is not parallelized; so that could be tackled. It would be great though to start with a baseline public shareable case.
a
I see - thanks! In our case we have two pex files that we unpack into the same venv using 
PEX_PATH=large.pex pex-tools small.pex venv /venv
. Not sure if this is intended behavior but it does merge the two pexes into the same venv. I'd rather unpack the large pex into a different 
venv-large
and reuse it because the large pex changes less frequently. Is it possible to "merge" two venvs at runtime?
e
Not right now - no. The merging is the intended behavior.
There are lots of ways to go about supporting this sort of thing - 3 come to mind: + Use 
PEX_EXTRA_SYS_PATH
+ some new feature to install a PEX's code in a flat dir (like a 
site-packages
dir without the surrounding venv). + Use 
PYTHONPATH=large.pex
+ `--non-hermetic-scripts`when installing the venv + add a shim 
import __pex__
somewhere early in the venv entrypoint code + Install 
large.pex
in a venv that uses 
--copies
, then point to that venv's 
python
when building the 
small.pex
venv and also enable 
--system-site-packages
The last I'm not sure would work.
The 1st two you can try out today. The 1st 1 is probably the most straight forward. It's ugly, but probably will work.
Lacking the new flat-dir feature, you just 
PEX_TOOLS=1
a venv for 
large.pex
and then 
PEX_EXTRA_SYS_PATH=/large/venv/lib/pythonX.Y/site-packages
when running the small pex venv.
Obviously it becomes on-you to make sure 
pythonX.Y
matches now between large and small.
a
thanks for the suggestions! using 
PEX_EXTRA_SYS_PATH=/large/venv/lib/pythonX.Y/site-packages
seems straightforward. i also need the console scripts but I could just add 
/large/venv/bin
to the 
PATH
.
e
Aha - and there you hit an issue!
Check out the shebang on those scripts.
The PEX_PATH approach does not have that issue.
So 
PEX_EXTRA_SYS_PATH
won't work, Python knows nothing of that. You'll need 
PYTHONPATH=large/site-packages:small/site-packages
+ the 
--non-hermetic-scripts
option to 
pex-tools ... venv
I think.
a
does running the 
/venv/small/pex
still respect the env 
PYTHONPATH
? not sure how isolated it tries to run.
e
It does not by default. You need to use the 
--non-hermetic-scripts
venv tool option to get rid of the 
-sE
in the python shebang there and in console scripts.
a
the console scripts don't seem to have 
-sE
anyway, but I do see a the 
-sE
removed from 
venv/pex
with 
--non-hermetic-scripts
e
Hrm, they should!
Are they scripts or console scripts?
For example:
Copy code
$ pex cowsay --include-tools -o cowsay.pex
$ PEX_VERBOSE=1 PEX_TOOLS=1 ./cowsay.pex venv here
pex: Laying out PEX zipfile /home/jsirois/dev/pantsbuild/jsirois-pex/cowsay.pex: 0.1ms
pex: Executing installed PEX for /home/jsirois/dev/pantsbuild/jsirois-pex/./cowsay.pex at /home/jsirois/.pex/unzipped_pexes/d4455ae19e928aac33b14fa15e6cfcf00dd81e6b
pex:   Testing /usr/bin/python3.10 can resolve PEX at /home/jsirois/.pex/unzipped_pexes/d4455ae19e928aac33b14fa15e6cfcf00dd81e6b: 1.6ms
pex: Using the current interpreter /usr/bin/python3.10 since it matches constraints and PYTHONPATH is not set.
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pex/here/bin/cowsay
pex: Executing PEX_TOOLS venv: 28.9ms
pex:   Laying out PEX zipfile /home/jsirois/dev/pantsbuild/jsirois-pex/cowsay.pex: 0.1ms
$ head -1 here/bin/cowsay
#!/home/jsirois/dev/pantsbuild/jsirois-pex/here/bin/python3.10 -sE
Whereas:
Copy code
$ PEX_TOOLS=1 ./cowsay.pex venv --force --non-hermetic-scripts here
$ head -1 here/bin/cowsay
#!/home/jsirois/dev/pantsbuild/jsirois-pex/here/bin/python3.10
As I understand it, scripts are inherently broken since they are not easily / automatically tied to the venv Python.
a
I used flask to test, but I can't repro the issue (I do see 
-sE
in 
bin/flask
). Possibly an error in testing.
e
Ah, ok. Good.
6 Views
Open in Slack
PreviousNext
Powered by