Related to my last Q ^ about

run

. Version is

2.13.0dev5

. We have a project that contains some

pex_binary

targets with default

execution_mode

. When I do, e.g.,

./pants run ./proj/app/oasmodel:skew

everything works. When another user does it (after pulling the same repo, using the same pants binary,

pants.toml

, etc.), the execution crashes from

ImportError

once a 3rdparty# dependency is encountered. The

skew.pex

are identical (particularly

./.deps

) on first inspection of

./.pands.d/...

. Furthermore,

pants package proj/app/oasmodel:skew && ./dist/proj.app.oasmodel/skew.pex

has the same issue. On the user it does not work for, we tried setting

execution_mode = "venv"

on the target, and

pants package proj/app/oasmodel:skew && ./dist/proj.app.oasmodel/skew.pex

worked. An even more surprising behavior than ^ occurred when we tried

pants run proj/app/oasmodel:skew

-- it worked! But comparing

./.pants.d/venv-mode-run

(venv execution mode) vs.

./pants.d/default-mode-run

(default execution mode), they look basically identical, and

.deps

folders both contain the 3rdparty dependencies. As said, the former fails from importing any module exposed by a 3rdparty dist and the latter succeeds. I would not have expected this behavior given that

run

does its own thing and doesn't care about

exceution_mode

attribute in

pex_binary

(or does it to some extent?). I think my next step is to look where the actual Pex got splayed out on the filesystem (I thought that would be in

.pants.d/

when using

run

, but I guess it's in

$PEX_ROOT/

?) and see how it mapped

skew.pex/.deps

Python distributions into actual importable (or lack thereof) modules. But based on what I've seen so far, it does seem like an issue with Pex execution (not Pex building as the builds look the same) -- are there obvious environment setups that would cause the conversion of dists in

.deps

into importable modules to fail? I am in the process of assembling the complete logs and looking at the process outputs more closely. I'll open an issue for that, but wanted to give a heads up here and see if this prompts any ideas. One thing that I am suspicious of is this user was using VSCode Server (Linux) / VS Code (Windows w/ SSH plugin), and in the output I saw lots of unexpected tmp folders Pex was logging w/

-vscode-

labels in the paths. I wonder if the VSCode is doing something to the environment when using the terminal inside the IDE (but I also tried Putty and got the same behavior, though there could be an

.env

file in the root causing the same behavior).

the field does impact some things, but only how 3rd-party dependencies are handled I'm not surprised that

venv

fixes things. That is what you should use for maximum compatibility with 3rd-party deps. Looks like https://github.com/pantsbuild/pants/pull/15849 will always use

venv

when `run`ning a

python_source

target

Indeed, this is 3rdparty dependency related. So I can see that the Pex file itself has the same set of 3rdparty dependencies in its

.deps

folder (the collection of

.whl

). But when the Pex is executed, apparently the resolution of those dists into importable modules fails for this user if execution mode != venv (but otherwise succeeds for everyone else's system we've tested on -- there is nothing complex about this project and its targets/dependencies that would suggest venv mode is needed).

The short of it is: the default zipapp mode uses mildly complex

sys.path

scrubbing to make sure the ambient Python Interpreter's environment does not pollute your PEXes runtime environment before importing your code and running it. That scrubbing code is clearly buggy for this one user's environment. With the venv mode though, there is no Pex runtime scrubbing, the Python venv isolation is trusted to work. This is both faster (on 2nd+ runs) and more robust, assuming Python venv isolation is robust - which it has proven to be.

Makes sense. I'll take a look at his homedir next and see if there's anything suspicious there. As all our environments are to some extent based on NFS homedir setup and given the inconsistencies observed user to user for a myriad of Pex/LMDB related things before we moved our dev environments (mostly) to disk, I’m not as surprised anymore given what's been pointed out here. Do you think we should just macro

pex_binary

to set

execution_mode = "venv"

? Is bringing out the "big guns" by default sane?

This shouldn't be related to NFS. Instead it would be related to a system Python install different from others. Usually, a system Python with packages installed using either

sudo pip install ...

or using the system package manager - say

apt

or

brew

.

@enough-analyst-54434 This person is on vacation this week, but with that in mind, I will take a look at his environment when he returns. There is no possibility he has a different Python env setup as a result of using sudo or any other package manager / root installs as we're just SSHing into the same host and don't have that sort of access. It is however possible he has some env setup in his homedir that is causing this issue to occur.