https://pantsbuild.org/ logo
#general
Title
# general
m

modern-leather-27612

10/06/2021, 8:41 PM
Hi. First, thanks for the work on Pants. In my proof-of-concept Pants-based monorepo for Enigma, I got to the point where I am configuring the CI pipeline for merge requests. Being able to realize my ideal testing strategy in this one-liner is pretty darn cool:
Copy code
./pants --changed-since=origin/boss --changed-dependees=transitive test
However, there is one area that has left me completely befuddled and I would much appreciate pointers to documentation I might have missed or an explanation. That area is PEX. When I run Python in a virtual environment, I can easily test for its presence (
sys.prefix != sys.base_prefix
) and I can easily configure the environment for a subprocess (prepend `bin`/`Scripts` to
PATH
, set
VIRTUAL_ENV
to root of venv). If I run
pip list
in that subprocess it will be the
pip
installed in the venv and it will correctly report what other packages are installed as well. Amazingly, I don't even have to include
PYTHONPATH
in the environment. Clearly, PEX does things differently. As much is obvious from my instrumentation. Notably, the proof-of-concept really is a trivial tool to collect relevant information from
sys
,
sysconfig
, and
os.environ
. It also extracts installed packages by spawning
pip
, using
importlib.metadata.distributions()
, and an evil hack that looks at the names and versions of the wheel files in
sys.path
. As you can see, I did try to do my homework. But try as I may, I'm getting very strange behavior when using
pip
. First, it runs a completely different, older pip than the one inside the PEX. Second, that pip reports no packages installed besides itself. Third, it maintains that this is the case even when I prime it with the different paths on
sys.path
. I am forwarding
sys.path
through
PYTHONPATH
and leaving the environment otherwise untouched. Uhm, do you have an idea what's going on? And how do I spawn subprocesses correctly from within a PEX? Thanks!
e

enough-analyst-54434

10/06/2021, 10:21 PM
If you only have the issue when running the PEX after creating it with
./pants package ...
then you should be able to get what you expect by setting your
pex_binary
target's
execution_mode
to `"venv"`: https://www.pantsbuild.org/docs/reference-pex_binary#codeexecution_modecode
m

modern-leather-27612

10/07/2021, 12:33 AM
Hi John, thanks! I missed that in the documentation. Though after trying to debug all this, I am still curious as to how this all works internally, even when not using venv mode. Is there documentation anywhere or a video or something? I started reading the sources but there is a lot of them and that makes the code hard to digest without some architectural guide.
e

enough-analyst-54434

10/07/2021, 12:43 AM
If you want to understand traditional (non venv) PEX isolation, there is only the code. That said, if you print sys.path and compare that with the contents of PEX-INFO for your PEX file (which is a zip), you can probably piece together all the pertinent bits from those two pieces of data alone.
m

modern-leather-27612

10/07/2021, 3:22 PM
Cool. That was on my todo list already, so now I'll make that exercise a priority. Thank you again!
I just updated my
BUILD
rule with
execution_mode="venv"
and also fixed the PIP-based inspection (there was a small bug, oops). I'm happy to confirm that PIP,
importlib.metadata.distributions()
, and my evil wheel-name-based hack all yield the same information. Plus the previous PIP-based inspector actually wasn't so far off either. The only problem there was that it somehow managed to load the outdated PIP that came with the Python runtime instead of the latest inside the PEX. But all good now.
❤️ 2
b

busy-vase-39202

10/13/2021, 6:50 PM
Aside: your one-liner soon will be able to get even shorter https://github.com/pantsbuild/pants/pull/13228
❤️ 1
4 Views