https://pantsbuild.org/ logo
Join Slack
Powered by
anyone have any current advice for how best to ach...
# pex
r
anyone have any current advice for how best to achieve pex-style env scrubbing via API? we want to bootstrap arbitrary pex environments in a jupyter notebook kernel w/ env scrubbing for hermeticity.
e
What do you mean by env? I'm guessing the full hermetic python interpreter suite of env vars, sys.path, packages with undesired path elements, etc - the full suite?
r
yep. we start the jupyter kernel from a pex cli invocation with a PEX_PATH chain of 2-3 other pex files. then we have pants and pex plugins in the notebook that are called as “magics”, e.g. 
%pants_load src/python/x/y:z
that builds a pex and then bootstraps in the live kernel via 
pex.pex_bootstrapper
.
so each time that plugin is invoked, ideally we’d scrub any previous pex-provided state from sys.path et al (from the initial launch and subsequent plugin invokes).
e
Ok. I'll look at this. One thing for sure, I'd love to get more tests and docs around the bootstrapper API. Just looking at the PEX codebase, ignorant of external usages, it's always scary. I'd love to kill it or formalize it a bit better.
r
that would be awesome
here’s a quick demo of what we’re working on
pex-pants-jupyter.mov
pex-pants-jupyter.mov
would love to open source this work at some point under pantsbuild github maybe? could be generally useful.
e
I definitely don't have enough context, but maybe. At a very dumb, high level (I've never used ipython or jupyter or interacted with multi-gig dependencies like the ML world has): would a "normal" user just have a venv they run jupyter in and in that video use pip install instead of %pex_magic ?
r
so in this case e.g 
%requirements_load
runs 
pex -f science-binaries --no-pypi <req> <req> etc
to net the same effect in a pex-ified world sans open network access.
(for adhoc loading cases)
and 
%pants_load
does the same at the pants layer for entire target closures.
but there’s nothing special about the sys.path/env setup in this case, it’s just as if you’ve loaded up a pex repl - then bootstrap another pex inside it.
e
Gotcha
r
here’s what I came up with for a v1 of pre-bootstrap scrub: https://gist.github.com/kwlzn/ec0a870eca9985a86ab1e468519cd29c
now calling this as earth scorcher + 
pex_bootstrapper.bootstrap_pex_env()
in succession - seems to work reasonably well, probably overlooking a few things in the import machinery tho but expecting most of that to get covered by bootstrap.
I found that a whitelist was needed to avoid scrubbing sys.modules to the point of breaking the actual notebook kernel, so it’s a bit leaky.
anyhow, good enough for the moment
3 Views
Open in Slack
PreviousNext
Powered by