Hey folks, I have a use case in Python that I suspect isn't terribly special, and I wonder if folks could point me in the right direction. I have a codebase with a large library, and a handful of applications that use the library. The final distribution target is a container image, and I'm hoping to build the application itself as a Pex. When we ship the final build, the modules in the library are cythonized to shared library files. We do this today via setuptools

build_ext

, and implement our own cache so that we only rebuild modules that have changed. I can define a distribution target for the library in Pants that triggers the

build_ext

routine, but in this case our cache doesn't work. And when building the Pex, I'd like to bring along just those shared library files that are required and not the whole boatload. It seems like what I want is to be able to define a target for each eventual shared library file, with dependencies looking something like:

pex -> application code -> .so file -> .py source

Before I dive too deep into the plugin framework, any pointers or feedback? If I were to go the route of writing a plugin, any words of advice for how to think about this?

welcome! 🙂 Let's brainstorm!

I can define a distribution target for the library in Pants that triggers the build_ext routine, but in this case our cache doesn't work.

Right, so you are thinking of

python_distribution

target either with an existing

setup.py

or a generated one. You say the cache doesn't work because when building a Python wheel, Pants will re-compile all your sources (using the

setup.py

tooling), am I understanding it right?

And when building the Pex, I'd like to bring along just those shared library files that are required and not the whole boatload.

So the PEX files are not related to the

python_distribution

target (your Python wheel), are they? The

pex_binary

target would only understand the Python sources, whereas you want to tell what

.so

files to stuff into the

.pex

file from the

build

directory produced when Python wheel was generated?

Hey! I’m thinking this should be a discussion on GitHub? https://github.com/pantsbuild/pants/discussions/new ?

Thanks for the replies! I'll start a thread over there and post a link here

@happy-kitchen-89482 Doesn't this relate to that modification you made some months ago, allowing customization of the setup.py on the fly?

I don’t recall allowing customization of a generated setup.py on the fly (DYM https://github.com/pantsbuild/pants/issues/12487?) but I’m not sure this would relate either way. I think to get the full benefit of invalidation and caching we’d want to model this outside of setup.py

Yeah, I should clarify that use of setup.py here is incidental; we just happen to be doing it that way now

I think to get the full benefit of invalidation and caching

Couldn't we still cache the output after setup.py? Either way, would love to see an official cython or mypyc solution. I feel like so much perf is left on the table otherwise.

Let’s discuss on the discussion 🙂