https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to specify the location of the `pex...
# general
q
Is there a way to specify the location of the 
pex
executable for 
pants_jupyter_plugin
instead of fetching it from GitHub? (ie. https://github.com/pantsbuild/pants-jupyter-plugin/blob/ae67015a7d011315973ccc4416fdabd115a79197/pants_jupyter_plugin/pex.py#L45-L50) We were attempting to use pex + pants_jupyter_plugin to isolate user notebook dependencies from the python environment used to launch jupyter notebook server in a deployment that was behind a heavily restricted firewall (that in particular was denying access to 
<http://github.com|github.com>
& 
*.<http://githubusercontent.com|githubusercontent.com>
), and the kernel failed to connect if the pex executable wasn't present on startup. Unfortunately we needed to stop using this in the near-term to unblock, but would love to figure out a path forward since using pex to isolate nb deps is pretty ideal for us
e
There is not as you can see from your code pointer. I'd be happy to review your contribution to plumb an option though and get a release out with your fix.
q
Happy to follow up with a contribution to plumb through another location!
e
Excellent.
q
May be a bit before we get around to it, but otherwise we love pants_jupyter_plugin.
e
Which magic do you use?
We haven't gotten usage feedback yet from non-Twitter users to my understanding and I'd love to know how you use it.
q
%pex_load mostly. We still are stuck using bazel in our monorepo but I recently wrote a bazel rule to build pexes from py_library/py_binary targets
We use pex to store “pickle compatible” environments for certain pieces of code (usually ML models that can't be serialized to a format like ONNX)
And this helps our research team when debugging behavior of older model versions
e
Aha. Ok. You may be intereted in modern Pex. A modern PEX can simply de imported from, just:
Copy code
sys.path.append("/the/pex")
import __pex__
import thing.in.pex
Or:
Copy code
sys.path.append("/the/pex")
from __pex__ import thing.in.pex
No special support is needed at all.
q
Oh interesting - that works with a zipped pex?
e
q
Ooh. Will check that out.
e
Only works if the PEX file was built by Pex >= 2.1.98 of course.
q
We haven't fully rolled this out yet so now would be the time to upgrade
e
Ok, if that works for you I'd also love to know.
q
Sure thing
@enough-analyst-54434 what happens to the sys.path for shared libraries between the host script & the PEX? For example, if we were using v2 of 
lib
in the script loading the pex, but the pex contained v1 of 
lib
, which version of 
lib
would be present in the environment after updating the `sys.path`/using the 
from __pex__
import
Hopefully will get to play around with this tomorrow or next week
e
That's actually a Python thing and not a Pex thing. Python will scan the sys.path in order. So if you 
sys.path.append("/path/to/pex")
the Pex will be looked in last by the Python import system, if you instead `sys.path.insert(0, "/path/to/pex")`it'll be asked 1st.
👍 1
But it will only be asked if the thing hasn't yet been imported. If it already has, the Python import system will stick with what it has in 
sys.modules
- the import cache.
👍 1
6 Views
Open in Slack
PreviousNext
Powered by