https://pantsbuild.org/ logo
Join Slack
Powered by
One of the Jupyter Lab modules is called `ipykerne...
# pex
j
One of the Jupyter Lab modules is called 
ipykernel
but it installs the module 
ipykernel_launcher
. Is there a way to teach the pex this alias or to force the "install" of particular wheels?
h
By module, do you mean distribution? As in, something installable via 
pip
?
j
Looks like the 
ipython...whl
installs 
IPython
which is an issue on macOS but might be a problem on Linux.
From `ls`:
Copy code
ipykernel-5.3.4-py3-none-any.whl:
ipykernel                 ipykernel-5.3.4.dist-info ipykernel_launcher.py     share
h
Is there a way to teach the pex this alias or to force the “install” of particular wheels?
You can use 
pex req1 req2
to ensure that both are installed. Alterantively, you can use a constraints file to force a particular version of a distribution to be used, even if that distribution is a transitive dep and not explicitly specified on the CLI. 
pex --constraints constraints.txt req1
j
The 
jupyterlab.labapp:main
function loads 
ipykernel_launcher.py
. Looking at the wheel, I am guessing it isn't finding it because it is in the 
ipykernel
wheel.
I used 
pex -r $(pip freeze)
in a virtualenv that had JupyterLab installed. So the wheel is there. I am not sure how 
pex
knows about all the different scripts and modules contained in the 
.pex
file.
h
When you install via Pex, Pex sets PYTHONPATH to include 
~/.pex/installed_wheels/my-dist-blah.whl/
, e.g. 
ipykernel-5.3.4-py3-none-any.whl
. This means that imports of things like 
ipykernel_launcher.py
should work
j
Maybe jupyter is doing it via an 
exec
or 
popen
. Let me eliminate that possibility. Thank you!
h
It could be, I think Jupyter does some non-standard stuff. FYI this is how we got iPython working: https://github.com/pantsbuild/pants/blob/969c8dcba6eda0c939918b3bc5157ca45099b4d1/src/python/pants/backend/python/subsystems/ipython.py#L7-12
j
That is helpful!
e
To circle back on concepts here: 1. The wheel prefix (ipykernel for ipykernel-5.4.3*.whl) is a normalized (lowercased, and a few other normalizations) form of the "project name". The project name need have zero correlation to contained module names. It's just that most projects have project name == root package / module. 2. Pex does not guess scripts or entry points. If you build a pex and don't specify -e or -c it will have no entry point and will just launch an interpreter in a ~venv containing only the stdlib and the code and wheels included in the PEX file.
To point 1. The pantsbuild.pants project / distributions contain a top level package is pants, not pantsbuild. Likewise, the setuptools project contains top level packages of setuptools and pkg_resources.
j
👍🏽
The problem is that Jupyter manages multiple processes at the same time. What it really needs is a venv where all its dependences are installed. Jupyter defines its kernels in 
kernel.json
files.
The ipython kernel is defined by default as 
$PYTHONEXE -m ipykernel_launcher
.
When I run my pex, the Jupyter Labs app seems to not pay attention to the pex and just seems to interact with a python on 
$PATH
.
e
I may have missed it, but can you provide the full command line you use right now? And if any -r the contents of those?
j
pex jupyterlab ipykernel -o jupyter-labs.pex -c jupyter-lab
I've also tried installing 
jupyterlab
into a virtenv and using the 
pip freeze
list. Same results.
This is what gets put into 
.deps
in the pex:
Copy code
Jinja2-2.11.2-py2.py3-none-any.whl                   nbclient-0.5.0-py3-none-any.whl
MarkupSafe-1.1.1-cp36-cp36m-macosx_10_6_intel.whl    nbconvert-6.0.7-py3-none-any.whl
Pygments-2.7.1-py3-none-any.whl                      nbformat-5.0.7-py3-none-any.whl
Send2Trash-1.5.0-py3-none-any.whl                    nest_asyncio-1.4.1-py3-none-any.whl
appnope-0.1.0-py2.py3-none-any.whl                   notebook-6.1.4-py3-none-any.whl
argon2_cffi-20.1.0-cp36-abi3-macosx_10_15_x86_64.whl packaging-20.4-py2.py3-none-any.whl
async_generator-1.10-py3-none-any.whl                pandocfilters-1.4.2-py3-none-any.whl
attrs-20.2.0-py2.py3-none-any.whl                    parso-0.7.1-py2.py3-none-any.whl
backcall-0.2.0-py2.py3-none-any.whl                  pexpect-4.8.0-py2.py3-none-any.whl
bleach-3.2.1-py2.py3-none-any.whl                    pickleshare-0.7.5-py2.py3-none-any.whl
certifi-2020.6.20-py2.py3-none-any.whl               prometheus_client-0.8.0-py2.py3-none-any.whl
cffi-1.14.3-2-cp36-cp36m-macosx_10_9_x86_64.whl      prompt_toolkit-3.0.8-py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl                   ptyprocess-0.6.0-py2.py3-none-any.whl
decorator-4.4.2-py2.py3-none-any.whl                 pycparser-2.20-py2.py3-none-any.whl
defusedxml-0.6.0-py2.py3-none-any.whl                pyparsing-2.4.7-py2.py3-none-any.whl
entrypoints-0.3-py2.py3-none-any.whl                 pyrsistent-0.17.3-cp36-cp36m-macosx_10_15_x86_64.whl
idna-2.10-py2.py3-none-any.whl                       python_dateutil-2.8.1-py2.py3-none-any.whl
importlib_metadata-2.0.0-py2.py3-none-any.whl        pyzmq-19.0.2-cp36-cp36m-macosx_10_9_intel.whl
ipykernel-5.3.4-py3-none-any.whl                     requests-2.24.0-py2.py3-none-any.whl
ipython-7.16.1-py3-none-any.whl                      setuptools-50.3.0-py3-none-any.whl
ipython_genutils-0.2.0-py2.py3-none-any.whl          six-1.15.0-py2.py3-none-any.whl
jedi-0.17.2-py2.py3-none-any.whl                     terminado-0.9.1-py3-none-any.whl
json5-0.9.5-py2.py3-none-any.whl                     testpath-0.4.4-py2.py3-none-any.whl
jsonschema-3.2.0-py2.py3-none-any.whl                tornado-6.0.4-cp36-cp36m-macosx_10_15_x86_64.whl
jupyter_client-6.1.7-py3-none-any.whl                traitlets-4.3.3-py2.py3-none-any.whl
jupyter_core-4.6.3-py2.py3-none-any.whl              urllib3-1.25.10-py2.py3-none-any.whl
jupyterlab-2.2.8-py3-none-any.whl                    wcwidth-0.2.5-py2.py3-none-any.whl
jupyterlab_pygments-0.1.2-py2.py3-none-any.whl       webencodings-0.5.1-py2.py3-none-any.whl
jupyterlab_server-1.2.0-py3-none-any.whl             zipp-3.3.0-py3-none-any.whl
mistune-0.8.4-py2.py3-none-any.whl
e
And what is the success crteria?
j
I want to be able to run the Jupyter Lab app as if I have launched it from a python virtualenv.
Copy code
mktmpenv
pip install jupyterlab
jupyter-labs
have the same results as running 
jupyter-labs.pex
.
The libraries install supporting files for the web server and client (
.js
etc).
e
After I build the pex with your exact command, I run the pex and get a browser open to:
Copy code
JupyterLab Error
JupyterLab application assets not found in "/usr/share/jupyter/lab"
Please run `jupyter lab build` or use a different app directory
You similar?
j
yes. I solved that by putting the shared files into a known path. Let me find the command I did for that.
cp -r share/jupyter /Users/myacct/.pyenv/versions/3.6.9/share/jupyter
This is a hack but would be resolved if I could just create a VirtualEnv on the run server. Maybe pex isn't the way to go.
h
Raúl, remind us, how are you getting this all working with your v1 plugin?
j
🙂
Let me put that on the web. I got permission to publish it but didn't do that yet.
❤️ 1
e
So, the pex works fine if:
Copy code
$ ./jupyter-labs.pex --app-dir=/home/jsirois/.pex/installed_wheels/9642a3f7a665b0297c417dee9da68c8db0892eaf/jupyterlab-2.2.8-py3-none-any.whl/share/jupyter/lab
The only issue then is jupyter can't find share/jupyter/lab when its present on sys.path in a non-standard location it seems.
j
e
Yes.
👍🏽 1
j
@enough-analyst-54434 When you used 
--app-dir
, were you able to open a console in the app and run python code? I don't think a kernel gets launched by default the first time you run.
e
Yes
Thats the default 
--app-dir
detection code.
j
Maybe we can take advantage of 
JUPYTERLAB_DIR
. Can ENV variables be set dynamically to the 
.pex/installed_wheels/9642a3f7a665b0297c417dee9da68c8db0892eaf
directory?
e
That is definitely only going to work with a venv out of the box. We'd need to add a shim main that found the 
jupyterlab
distribution on sys.path and then exported 
JUPYTERLAB_DIR=...
that.
👍🏽 1
j
I think making the venv is the safer way to go. That seems to fit the design philosophy of the Jupiter ecosystem.
Pants builds venv.
I think the pex step may be in the wrong direction.
What I really want for my plugin is to create a venv with jupyter labs AND my targets. Then launch the notebook app.
e
That should also have failed in exactly the same way in the v1 plugin you pointed to in the gist.
The only difference in v2 it your'll be creating a PEX file from the CLI. In v1 you created a PEX loose directory via the Pex API.
👍 1
j
I think the 
jupyter_entry_point
is the fix for that.
Let me add that to the gist in more detail.
e
So, why in v2 aren't you using the same entrypoiny?
You're using 
-c jupyterlab
j
I don't know. I thought with the new v2 plugin API I could incorporate the requirements into the plugin without needing the extra code.
e
So, backing up. In v1 you build a Pex and it apparently works. In v2 you should do exactly the same and it should work the same.
The only difference in v1 is you had access to the Pex API. In v2 you do not, you should use the Pex CLI. Otherwise the pexes should work the same. You might need to add --not-zip-safe to the v2 version, but that's it.
SO, checking my 1 assumption there - the v1 version works like you'd like? I'm assuing that in the above.
j
Yes. v1 let's us type 
./pants jupyter project/src/python/projectapp
and get a notebook that can import 
projectapp
.
e
So, a difference here is the jupyterlab version. That's proved by the fact that your default v1 entrypoint no longer exists.
j
Is our v1 plugin injecting the Jupyter requirements into the pex that is building the target requirements?
e
Can you make sure you specify the same jupyter==VERSION on the pex cli?
Ah, yes you add
Copy code
jupyter_stubber
So your pex cli example above should be doing exactly the same.
j
Yes. I can. An upgrade to newer version is not needed on the first v2 plugin.
e
OK. To emphasize: for sanity sake it seems to me creating a PEX exactly like the working v1 plugin does is a 1st best place to start. Looking at the gist, that plugin has 1 requirement and that's `
Copy code
jupyter_stubber==1.0.0
👍🏽 1
j
When I first tried my v2 plugin I worked with the original code. But I did not know how to map all the v1 types and rules to new v2 types and rules.
e
I find:
Copy code
$ pex jupyter_stubber==1.0.0 -o jupyter-labs.pex -e jupyter_stubber.stubber:run --not-zip-safe
ERROR: Could not find a version that satisfies the requirement jupyter_stubber==1.0.0 (from versions: none)
ERROR: No matching distribution found for jupyter_stubber==1.0.0
j
jupyter_stubber
is from our repo.
Let me get that code into the gist.
e
OK, ty.
But you can just try that.
When I first tried my v2 plugin I worked with the original code. But I did not know how to map all the v1 types and rules to new v2 types and rules.
So that makes sense. What I'm trying to point out is the Pants APIs aren't super relevant at the most basic level. At the most basic level you need to be able to create the same Pex you were creating in v1 via the Pex CLI. If you can do that it should work the same as the v1 Pex. At that point, and only at that point, you'll have apples to apples and you can go on to figuring out the new Pants APIs to use the Pex CLI.
👍🏽 1
j
updated
Our old 
pants.ini
has
Copy code
jupyter_requirements: ["jupyter_stubber==1.0.0"]
e
Does what I'm saying make sense though? The PEX cli you started with above is just plain different from the v1 plugin and that's not a great place to start.
j
That makes sense..
e
Cool.
j
With v2 plugins, will I be able to access the code in the repo the way we do with v1? Or do I need to teach my plugin how to build this exact same pex (
jupyter_stubber
)?
Forget that question. Let me build the v1 pex. And get that into my head.
Then I'll worry about how to build it with pants 2.0.0 for the v2 plugin.
e
Sounds good.
Open in Slack
PreviousNext
Powered by