Hi. First, thanks for the work on Pants. In my proof-of-concept Pants-based monorepo for Enigma, I got to the point where I am configuring the CI pipeline for merge requests. Being able to realize my ideal testing strategy in this one-liner is pretty darn cool:

./pants --changed-since=origin/boss --changed-dependees=transitive test

However, there is one area that has left me completely befuddled and I would much appreciate pointers to documentation I might have missed or an explanation. That area is PEX. When I run Python in a virtual environment, I can easily test for its presence (

sys.prefix != sys.base_prefix

) and I can easily configure the environment for a subprocess (prepend `bin`/`Scripts` to

PATH

, set

VIRTUAL_ENV

to root of venv). If I run

pip list

in that subprocess it will be the

pip

installed in the venv and it will correctly report what other packages are installed as well. Amazingly, I don't even have to include

PYTHONPATH

in the environment. Clearly, PEX does things differently. As much is obvious from my instrumentation. Notably, the proof-of-concept really is a trivial tool to collect relevant information from

sys

,

sysconfig

, and

os.environ

. It also extracts installed packages by spawning

pip

, using

importlib.metadata.distributions()

, and an evil hack that looks at the names and versions of the wheel files in

sys.path

. As you can see, I did try to do my homework. But try as I may, I'm getting very strange behavior when using

pip

. First, it runs a completely different, older pip than the one inside the PEX. Second, that pip reports no packages installed besides itself. Third, it maintains that this is the case even when I prime it with the different paths on

sys.path

. I am forwarding

sys.path

through

PYTHONPATH

and leaving the environment otherwise untouched. Uhm, do you have an idea what's going on? And how do I spawn subprocesses correctly from within a PEX? Thanks!

If you only have the issue when running the PEX after creating it with

./pants package ...

then you should be able to get what you expect by setting your

pex_binary

target's

execution_mode

to `"venv"`: https://www.pantsbuild.org/docs/reference-pex_binary#codeexecution_modecode

Hi John, thanks! I missed that in the documentation. Though after trying to debug all this, I am still curious as to how this all works internally, even when not using venv mode. Is there documentation anywhere or a video or something? I started reading the sources but there is a lot of them and that makes the code hard to digest without some architectural guide.

If you want to understand traditional (non venv) PEX isolation, there is only the code. That said, if you print sys.path and compare that with the contents of PEX-INFO for your PEX file (which is a zip), you can probably piece together all the pertinent bits from those two pieces of data alone.

Cool. That was on my todo list already, so now I'll make that exercise a priority. Thank you again!

Aside: your one-liner soon will be able to get even shorter https://github.com/pantsbuild/pants/pull/13228