https://pantsbuild.org/ logo
Join Slack
Powered by
I have a question about `import __pex__`. I have a...
# general
s
I have a question about 
import __pex__
. I have a minimal setup via (while this question is about pex, I'm interested in being able to consume `pex`s generated by pants. This just seemed like a more simplified way to reframe things)
Copy code
python3 -m venv venv
source venv/bin/activate
python -m pip install pex
python -m pip install ipython
python -m pex 'flask' -o flask.pex
python -m IPython

In [1]: import sys
In [2]: sys.path.append('flask.pex')
In [3]: import __pex__
but get
Copy code
ModuleNotFoundError: No module named 'prompt_toolkit'
Error in sys.excepthook:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ModuleNotFoundError: No module named 'IPython'
I have 
pex==2.1.145
installed
So after posting this, I realized I need to include ipython as part of the pex e.g. 
python -m pex flask ipython -o flask.pex
Unfortunately 
ipython
seems to get stuck
replacing ipython for jupyter and I get 
ModuleNotFoundError: No module named 'zmq'
when trying to run the above commands in jupyter
Going back to 
python -m pex 'flask' -o flask.pex
and running the following (lets call it 
main.py
)
Copy code
import sys
sys.path.append('flask.pex')
import __pex__
from flask import Flask
print(Flask)
works. Commands also work in 
python
.
I wonder what it is about 
ipython
that causes it to crash
e
Repls like IPython and Jupyter often make assumptions about how they're run. If you build a PEX with --venv , that tends to help the PEX conform to those assumptions.
s
Thanks, that got me a little farther.
Copy code
python3.10 -m pex flask --venv -o flask.pex
and spinning up 
python3.10 -m IPython
Copy code
In [1]: import sys

In [2]: sys.path.append('flask.pex')

In [3]: import __pex__
works. But when I try to 
import flask
I get 
ModuleNotFoundError: No module named 'hmac'
I'd say this is unrelated to pex, but importing 
flask
after 
import __pex__
works in a python script.
Well this is odd. Again from 
python3.10 -m IPython
Copy code
In [1]: import hmac

In [2]: import secrets

In [3]: import sys

In [4]: sys.path.append('flask.pex')

In [5]: import __pex__

In [6]: import flask

In [7]:
On the otherhand
Copy code
In [1]: import sys

In [2]: sys.path.append('flask.pex')

In [3]: import __pex__

In [4]: import hmac
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[4], line 1
----> 1 import hmac

ModuleNotFoundError: No module named 'hmac'
So I tried running the 
main.py
script again but I get a similar error this time. The difference being this one uses a PEX created with 
--venv
I tried 
--venv-copies
and 
--venv-site-packages-copies
and while those worked when running 
main.py
they don't work in 
ipython
.
e
@silly-queen-7197 before I dive in here, can you reveal your driving goal in selecting what goes in the venv vs what goes in the PEX? Do you not actually care, or do you care about the split, and if so, towards what end?
s
The end goal is to be able to create a 
pex
that captures dependencies (1st and 3rd party) from our monorepo that folks can consume in 
jupyter
notebooks.
We use 
vs code
and the embedded 
jupyter notebook
extension which will install necessary dependencies into a python environment so as we pick a 
python
version that satisfies our constraints I think we're set there. I just want to be able to declaratively create `pex`s like
Copy code
pex_binary(
    dependencies=["//:reqs#Flask"],
    execution_mode="venv"
)
that will be consumed via notebooks (or one giant 
pex
, I don't know if that's a better or worse approach)
In this setting if I do
Copy code
import sys
import os
sys.path.append('../dist/foobar/foobar.pex')
import __pex__
in the jupyter notebook (spawned from some place like 
~/.pyenv/versions/3.10.8/bin/python
not a 
venv
or 
pex
) I get 
ModuleNotFoundError: No module named 'secrets'
. Ultimately this is the error I really care about. Sorry about asking such broad questions! I really appreciate you taking the time to look at.
For what it's worth,
Copy code
pex_binary(
    dependencies=["//:reqs#Flask"],
)
works. I think what I'm confused about is I don't know if that's a bug or expected behavior
e
Can you tighten things up a bit? Works as in adding 
execution_mode="venv"
causes failure, but without that bit, you can:
Copy code
import sys
import os
sys.path.append('../dist/foobar/foobar.pex')
import __pex__
And not get the 
secrets
error?
What would be great is a summary that says something like: I'm using Jupyter==x.y.x and I run Z to start it. I then try to do these things with a PEX built for interpreter W and requirements A,B,C.
Give me a repro with steps in other words.
s
Can do
e
Thank you.
I know for ~certain a PEX with everything, including jupyter, has worked in the past; so this variant, jupyter supplied by a venv, PEX loaded in via 
__pex__
magic hook will involve some fiddly corner case that a repro will be useful for.
s
e
Ok, great. That looks perfect. I may or may not find time between now and the 14th, but will be back at a keyboard on the 15th. So worst case I should have something to say then.
s
Of course! So far I've been really enjoying pants / pex and it's been a big productivity multiplier. Thanks again
Looks like I had python 3.10.13 not 3.10.8 on this particular machine, but I don't think that matters. I decided to give debugging this a shot. Here's what I found
Copy code
mkdir with-venv
unzip with-venv/with-venv.pex -d with-venv/
To debug, I'm running the following in a notebook with the same setup as described in the README.md
Copy code
import sys
sys.path.append('./with-venv')
import __pex__
I set a breakpoint at 
import __pex__
. I can 
import secrets
at [pex_bootstrapper.py#L671](https://github.com/pantsbuild/pex/blob/main/pex/pex_bootstrapper.py#L671) however if I restart the notebook and run the debugger again, I can't import secrets at [pex_bootstrapper.py#L673](https://github.com/pantsbuild/pex/blob/main/pex/pex_bootstrapper.py#L673). If I inspect 
venv_python
I get 
/home/yjabri/.pex/venvs/6c8eaec5b07b6fe5d8c257cd72d1297dfafdfc8e/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/bin/python
. As a sanity check I can run
Copy code
/home/yjabri/.pex/venvs/6c8eaec5b07b6fe5d8c257cd72d1297dfafdfc8e/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/bin/python -c "import secrets"
without any exceptions. The line changes 
sys.path
from
Copy code
python
[
    "/home/yjabri/Projects/debug-pex-import/with-venv/.bootstrap/pex/vendor/_vendored/attrs",
    "/home/yjabri/Projects/debug-pex-import/with-venv/.bootstrap",
    "/home/yjabri/Projects/debug-pex-import",
    "/home/yjabri/.pyenv/versions/3.10.13/lib/python310.zip",
    "/home/yjabri/.pyenv/versions/3.10.13/lib/python3.10",
    "/home/yjabri/.pyenv/versions/3.10.13/lib/python3.10/lib-dynload",
    "",
    "/home/yjabri/.local/lib/python3.10/site-packages",
    "/home/yjabri/.pyenv/versions/3.10.13/lib/python3.10/site-packages",
    "./with-venv"
]
to
Copy code
python
[
    '/home/yjabri/.pex/venvs/6c8eaec5b07b6fe5d8c257cd72d1297dfafdfc8e/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/lib/python3.10/site-packages',
    './with-venv'
]
If I create a notebook with
Copy code
import sys
sys.path = [
    '/home/yjabri/.pex/venvs/6c8eaec5b07b6fe5d8c257cd72d1297dfafdfc8e/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/lib/python3.10/site-packages',
    './with-venv'
]
Then 
import secrets
throws a 
ModuleNotFoundError
. However if I modify the path to
Copy code
import sys
sys.path = [
    '/home/yjabri/.pyenv/versions/3.10.13/lib/python3.10',
    '/home/yjabri/.pex/venvs/6c8eaec5b07b6fe5d8c257cd72d1297dfafdfc8e/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/lib/python3.10/site-packages',
    './with-venv'
]
then 
import secrets
works
Slack doesn't do a great job at rendering these things. I'll create an issue on github
I created https://github.com/pantsbuild/pex/issues/2235. Tried to avoid language like "works" and "doesn't work" and ensure I explain my inputs and outputs
Open in Slack
PreviousNext
Powered by