Hello. I was working on an optimization for V2 `ru...
# pex
Hello. I was working on an optimization for V2
so that source file changes don’t require rebuilding the PEX. That is, we would stop bundling
into the PEX itself, and only populate it in the chroot and run the requirements PEX against those loose files. In command line terms, something like running
$ ./requirements.pex app.py
. I couldn’t figure it out. Issues with discovering the module, and I think not setting the
properly. Should this be possible to generalize for any arbitrary set of source files + arbitrary entry point?
In command line terms, given:
Copy code
$ pex requests -o requests.pex
$ mkdir -p a/package
$ echo 'import requests; print(requests.__file__)' > a/package/module.py
You do this like so:
Copy code
$ PYTHONPATH=. PEX_INHERIT_PATH=fallback PEX_MODULE=a.package.module ./requests.pex
Note the unfortunate leak though.
👍 1
I think what you'd need to do is have run consume
and then have backends that support a binary optimized for local runs produce something special - pex could create an otherwise empty PEX_PATH pex that composes a requirements PEX and sources PEX and backends that don't support this could just provide an identity rule that
and outputs an unaltered
If there is a way to default the identity rule that would be great.
👍 1
Perhaps the signal could come from the BinaryFieldSet union types to avoid the identity rule. If the BinaryFieldSet instance flags special, the run rule can ask for
instead of
Agreed with a special RunnableBinary. With the pythonpath issue, the problem is worse hermiticity, right? Probably worth the trade off for much faster iteration speed
Correct, I'm not sure I agree on the conclusion. Now your run will use random things installed in your system python interpreter site-packages if they have the same leading module / package name as anything in the PEX requirements.
My example above showed that case.
If pex had a PEX_INHERIT_PYTHONPATH as distinct from PEX_INHERIT_PATH (means sys.path which happens to include PYTHONPATH entries), that would be safe.
👍 1
What do you think about us adding that? We recently had to use PEX_INHERIT_PATH to get Pylint source plugins working, and it would be great to avoid the issue there too.
Did you try
for the pylint issue?
Pylint shouldn't know the difference between loose sources and --unzip.
But that aside - maybe? More limited would be to include . in the sys.path which the python interpreter does. Since it's intended that a PEX file acts like a (hermetic) python interpreter - perhaps just adding that limited bit makes more sense.
I did not. I’m afk but see https://github.com/pantsbuild/pants/pull/9792 for context