https://pantsbuild.org/ logo
Powered by
#pex
Title
h

hundreds-father-404

05/27/2020, 12:00 AM
Hello. I was working on an optimization for V2 
run
so that source file changes don’t require rebuilding the PEX. That is, we would stop bundling 
sources
into the PEX itself, and only populate it in the chroot and run the requirements PEX against those loose files. In command line terms, something like running 
$ ./requirements.pex app.py
. I couldn’t figure it out. Issues with discovering the module, and I think not setting the 
entry_point
properly. Should this be possible to generalize for any arbitrary set of source files + arbitrary entry point?
e

enough-analyst-54434

05/27/2020, 12:27 AM
In command line terms, given:
Copy code
$ pex requests -o requests.pex
$ mkdir -p a/package
$ echo 'import requests; print(requests.__file__)' > a/package/module.py
You do this like so:
Copy code
$ PYTHONPATH=. PEX_INHERIT_PATH=fallback PEX_MODULE=a.package.module ./requests.pex
/usr/lib/python3.8/site-packages/requests/__init__.py
Note the unfortunate leak though.
👍 1
I think what you'd need to do is have run consume 
RunnableBinary
and then have backends that support a binary optimized for local runs produce something special - pex could create an otherwise empty PEX_PATH pex that composes a requirements PEX and sources PEX and backends that don't support this could just provide an identity rule that 
Get[CreatedBinary]
and outputs an unaltered 
RunnableBinary
If there is a way to default the identity rule that would be great.
👍 1
Perhaps the signal could come from the BinaryFieldSet union types to avoid the identity rule. If the BinaryFieldSet instance flags special, the run rule can ask for 
RunnbaleBinary
instead of 
CreatedBinary
...
h

hundreds-father-404

05/27/2020, 12:42 AM
Agreed with a special RunnableBinary. With the pythonpath issue, the problem is worse hermiticity, right? Probably worth the trade off for much faster iteration speed
e

enough-analyst-54434

05/27/2020, 12:47 AM
Correct, I'm not sure I agree on the conclusion. Now your run will use random things installed in your system python interpreter site-packages if they have the same leading module / package name as anything in the PEX requirements.
My example above showed that case.
If pex had a PEX_INHERIT_PYTHONPATH as distinct from PEX_INHERIT_PATH (means sys.path which happens to include PYTHONPATH entries), that would be safe.
👍 1
h

hundreds-father-404

05/27/2020, 12:50 AM
What do you think about us adding that? We recently had to use PEX_INHERIT_PATH to get Pylint source plugins working, and it would be great to avoid the issue there too.
e

enough-analyst-54434

05/27/2020, 12:52 AM
Did you try 
--unzip
for the pylint issue?
Pylint shouldn't know the difference between loose sources and --unzip.
But that aside - maybe? More limited would be to include . in the sys.path which the python interpreter does. Since it's intended that a PEX file acts like a (hermetic) python interpreter - perhaps just adding that limited bit makes more sense.
h

hundreds-father-404

05/27/2020, 12:54 AM
I did not. I’m afk but see https://github.com/pantsbuild/pants/pull/9792 for context
Powered by