https://pantsbuild.org/ logo
#general
Title
# general
a

abundant-autumn-67998

02/27/2023, 6:23 PM
I have a question about pex venvs and hard links. We unpack the same large pex file into multiple venvs on the same machine, and I want to make sure I'm doing everything possible to make the venv unpacking is fast and efficient. Do I understand correctly that the pex file itself is unpacked into PEX_ROOT once and the same package files are then shared via hard links into every venv that is created from that pex file? Is there anything I can do to make the second
pex-tools my-pex.pex venv /here
faster?
e

enough-analyst-54434

02/27/2023, 6:29 PM
You understand correctly, but I also encourage you to check using
stat
! There are no low-hanging fruit to speed up venv creation that I'm aware of, but it is not parallelized; so that could be tackled. It would be great though to start with a baseline public shareable case.
a

abundant-autumn-67998

02/28/2023, 5:10 PM
I see - thanks! In our case we have two pex files that we unpack into the same venv using
PEX_PATH=large.pex pex-tools small.pex venv /venv
. Not sure if this is intended behavior but it does merge the two pexes into the same venv. I'd rather unpack the large pex into a different
venv-large
and reuse it because the large pex changes less frequently. Is it possible to "merge" two venvs at runtime?
e

enough-analyst-54434

02/28/2023, 5:12 PM
Not right now - no. The merging is the intended behavior.
There are lots of ways to go about supporting this sort of thing - 3 come to mind: + Use
PEX_EXTRA_SYS_PATH
+ some new feature to install a PEX's code in a flat dir (like a
site-packages
dir without the surrounding venv). + Use
PYTHONPATH=large.pex
+ `--non-hermetic-scripts`when installing the venv + add a shim
import __pex__
somewhere early in the venv entrypoint code + Install
large.pex
in a venv that uses
--copies
, then point to that venv's
python
when building the
small.pex
venv and also enable
--system-site-packages
The last I'm not sure would work.
The 1st two you can try out today. The 1st 1 is probably the most straight forward. It's ugly, but probably will work.
Lacking the new flat-dir feature, you just
PEX_TOOLS=1
a venv for
large.pex
and then
PEX_EXTRA_SYS_PATH=/large/venv/lib/pythonX.Y/site-packages
when running the small pex venv.
Obviously it becomes on-you to make sure
pythonX.Y
matches now between large and small.
a

abundant-autumn-67998

02/28/2023, 7:01 PM
thanks for the suggestions! using
PEX_EXTRA_SYS_PATH=/large/venv/lib/pythonX.Y/site-packages
seems straightforward. i also need the console scripts but I could just add
/large/venv/bin
to the
PATH
.
e

enough-analyst-54434

02/28/2023, 7:01 PM
Aha - and there you hit an issue!
Check out the shebang on those scripts.
The PEX_PATH approach does not have that issue.
So
PEX_EXTRA_SYS_PATH
won't work, Python knows nothing of that. You'll need
PYTHONPATH=large/site-packages:small/site-packages
+ the
--non-hermetic-scripts
option to
pex-tools ... venv
I think.
a

abundant-autumn-67998

02/28/2023, 7:11 PM
does running the
/venv/small/pex
still respect the env
PYTHONPATH
? not sure how isolated it tries to run.
e

enough-analyst-54434

02/28/2023, 7:14 PM
It does not by default. You need to use the
--non-hermetic-scripts
venv tool option to get rid of the
-sE
in the python shebang there and in console scripts.
a

abundant-autumn-67998

02/28/2023, 7:41 PM
the console scripts don't seem to have
-sE
anyway, but I do see a the
-sE
removed from
venv/pex
with
--non-hermetic-scripts
e

enough-analyst-54434

02/28/2023, 7:50 PM
Hrm, they should!
Are they scripts or console scripts?
For example:
Copy code
$ pex cowsay --include-tools -o cowsay.pex
$ PEX_VERBOSE=1 PEX_TOOLS=1 ./cowsay.pex venv here
pex: Laying out PEX zipfile /home/jsirois/dev/pantsbuild/jsirois-pex/cowsay.pex: 0.1ms
pex: Executing installed PEX for /home/jsirois/dev/pantsbuild/jsirois-pex/./cowsay.pex at /home/jsirois/.pex/unzipped_pexes/d4455ae19e928aac33b14fa15e6cfcf00dd81e6b
pex:   Testing /usr/bin/python3.10 can resolve PEX at /home/jsirois/.pex/unzipped_pexes/d4455ae19e928aac33b14fa15e6cfcf00dd81e6b: 1.6ms
pex: Using the current interpreter /usr/bin/python3.10 since it matches constraints and PYTHONPATH is not set.
pex: Re-writing /home/jsirois/dev/pantsbuild/jsirois-pex/here/bin/cowsay
pex: Executing PEX_TOOLS venv: 28.9ms
pex:   Laying out PEX zipfile /home/jsirois/dev/pantsbuild/jsirois-pex/cowsay.pex: 0.1ms
$ head -1 here/bin/cowsay
#!/home/jsirois/dev/pantsbuild/jsirois-pex/here/bin/python3.10 -sE
Whereas:
Copy code
$ PEX_TOOLS=1 ./cowsay.pex venv --force --non-hermetic-scripts here
$ head -1 here/bin/cowsay
#!/home/jsirois/dev/pantsbuild/jsirois-pex/here/bin/python3.10
As I understand it, scripts are inherently broken since they are not easily / automatically tied to the venv Python.
a

abundant-autumn-67998

02/28/2023, 9:46 PM
I used flask to test, but I can't repro the issue (I do see
-sE
in
bin/flask
). Possibly an error in testing.
e

enough-analyst-54434

02/28/2023, 9:59 PM
Ah, ok. Good.
6 Views