https://pantsbuild.org/ logo
#general
Title
# general
n

nice-florist-55958

06/17/2022, 5:56 PM
Related to my last Q ^ about
run
. Version is
2.13.0dev5
. We have a project that contains some
pex_binary
targets with default
execution_mode
. When I do, e.g.,
./pants run ./proj/app/oasmodel:skew
everything works. When another user does it (after pulling the same repo, using the same pants binary,
pants.toml
, etc.), the execution crashes from
ImportError
once a 3rdparty# dependency is encountered. The
skew.pex
are identical (particularly
./.deps
) on first inspection of
./.pands.d/...
. Furthermore,
pants package proj/app/oasmodel:skew && ./dist/proj.app.oasmodel/skew.pex
has the same issue. On the user it does not work for, we tried setting
execution_mode = "venv"
on the target, and
pants package proj/app/oasmodel:skew && ./dist/proj.app.oasmodel/skew.pex
worked. An even more surprising behavior than ^ occurred when we tried
pants run proj/app/oasmodel:skew
-- it worked! But comparing
./.pants.d/venv-mode-run
(venv execution mode) vs.
./pants.d/default-mode-run
(default execution mode), they look basically identical, and
.deps
folders both contain the 3rdparty dependencies. As said, the former fails from importing any module exposed by a 3rdparty dist and the latter succeeds. I would not have expected this behavior given that
run
does its own thing and doesn't care about
exceution_mode
attribute in
pex_binary
(or does it to some extent?). I think my next step is to look where the actual Pex got splayed out on the filesystem (I thought that would be in
.pants.d/
when using
run
, but I guess it's in
$PEX_ROOT/
?) and see how it mapped
skew.pex/.deps
Python distributions into actual importable (or lack thereof) modules. But based on what I've seen so far, it does seem like an issue with Pex execution (not Pex building as the builds look the same) -- are there obvious environment setups that would cause the conversion of dists in
.deps
into importable modules to fail?
I am in the process of assembling the complete logs and looking at the process outputs more closely. I'll open an issue for that, but wanted to give a heads up here and see if this prompts any ideas. One thing that I am suspicious of is this user was using VSCode Server (Linux) / VS Code (Windows w/ SSH plugin), and in the output I saw lots of unexpected tmp folders Pex was logging w/
-vscode-
labels in the paths. I wonder if the VSCode is doing something to the environment when using the terminal inside the IDE (but I also tried Putty and got the same behavior, though there could be an
.env
file in the root causing the same behavior).
h

hundreds-father-404

06/17/2022, 6:00 PM
the field does impact some things, but only how 3rd-party dependencies are handled I'm not surprised that
venv
fixes things. That is what you should use for maximum compatibility with 3rd-party deps. Looks like https://github.com/pantsbuild/pants/pull/15849 will always use
venv
when `run`ning a
python_source
target
n

nice-florist-55958

06/17/2022, 6:08 PM
Indeed, this is 3rdparty dependency related. So I can see that the Pex file itself has the same set of 3rdparty dependencies in its
.deps
folder (the collection of
.whl
). But when the Pex is executed, apparently the resolution of those dists into importable modules fails for this user if execution mode != venv (but otherwise succeeds for everyone else's system we've tested on -- there is nothing complex about this project and its targets/dependencies that would suggest venv mode is needed).
👍 1
e

enough-analyst-54434

06/17/2022, 9:02 PM
The short of it is: the default zipapp mode uses mildly complex
sys.path
scrubbing to make sure the ambient Python Interpreter's environment does not pollute your PEXes runtime environment before importing your code and running it. That scrubbing code is clearly buggy for this one user's environment. With the venv mode though, there is no Pex runtime scrubbing, the Python venv isolation is trusted to work. This is both faster (on 2nd+ runs) and more robust, assuming Python venv isolation is robust - which it has proven to be.
👍 1
n

nice-florist-55958

06/17/2022, 9:10 PM
Makes sense. I'll take a look at his homedir next and see if there's anything suspicious there. As all our environments are to some extent based on NFS homedir setup and given the inconsistencies observed user to user for a myriad of Pex/LMDB related things before we moved our dev environments (mostly) to disk, I’m not as surprised anymore given what's been pointed out here. Do you think we should just macro
pex_binary
to set
execution_mode = "venv"
? Is bringing out the "big guns" by default sane?
(So I guess not really worth opening up an issue report—could just be added to the list being compiled on the NFS-related open issue from awhile back?)
e

enough-analyst-54434

06/17/2022, 9:53 PM
This shouldn't be related to NFS. Instead it would be related to a system Python install different from others. Usually, a system Python with packages installed using either
sudo pip install ...
or using the system package manager - say
apt
or
brew
.
👍 1
n

nice-florist-55958

06/27/2022, 3:39 PM
@enough-analyst-54434 This person is on vacation this week, but with that in mind, I will take a look at his environment when he returns. There is no possibility he has a different Python env setup as a result of using sudo or any other package manager / root installs as we're just SSHing into the same host and don't have that sort of access. It is however possible he has some env setup in his homedir that is causing this issue to occur.
6 Views