thousands-sandwich-2635
12/05/2022, 7:50 PMbuild_ext
, and implement our own cache so that we only rebuild modules that have changed. I can define a distribution target for the library in Pants that triggers the build_ext
routine, but in this case our cache doesn't work. And when building the Pex, I'd like to bring along just those shared library files that are required and not the whole boatload.
It seems like what I want is to be able to define a target for each eventual shared library file, with dependencies looking something like:
pex -> application code -> .so file -> .py source
Before I dive too deep into the plugin framework, any pointers or feedback? If I were to go the route of writing a plugin, any words of advice for how to think about this?fresh-cat-90827
12/05/2022, 11:21 PMI can define a distribution target for the library in Pants that triggers the build_ext routine, but in this case our cache doesn't work.Right, so you are thinking of
python_distribution
target either with an existing setup.py
or a generated one. You say the cache doesn't work because when building a Python wheel, Pants will re-compile all your sources (using the setup.py
tooling), am I understanding it right?
And when building the Pex, I'd like to bring along just those shared library files that are required and not the whole boatload.So the PEX files are not related to the
python_distribution
target (your Python wheel), are they? The pex_binary
target would only understand the Python sources, whereas you want to tell what .so
files to stuff into the .pex
file from the build
directory produced when Python wheel was generated?happy-kitchen-89482
12/05/2022, 11:43 PMthousands-sandwich-2635
12/05/2022, 11:56 PMwide-midnight-78598
12/06/2022, 12:25 AMhappy-kitchen-89482
12/06/2022, 12:52 AMthousands-sandwich-2635
12/06/2022, 12:53 AMwide-midnight-78598
12/06/2022, 1:08 AMI think to get the full benefit of invalidation and cachingCouldn't we still cache the output after setup.py? Either way, would love to see an official cython or mypyc solution. I feel like so much perf is left on the table otherwise.
happy-kitchen-89482
12/06/2022, 3:19 AM