It’s been a while since I last looked into pants. ...
# general
e
It’s been a while since I last looked into pants. What is the current status of support for compiling extension modules w/ numpy distutils? Alternatively, can pants support arbitrary subprojects which use
setup.py
rather than BUILD files? I could probably build a wheel of the subproject and have pants treat it is an external dependency, but would appreciate if pants could automate this for me.
h
Hello! Welcome back, Pants has seen a lot of major improvements with the 2.0 launch, including dependency inference to shrink BUILD files by 90% on average. Much less boilerplate now. https://blog.pantsbuild.org/introducing-pants-v2/
What is the current status of support for compiling extension modules w/ numpy distutils?
I want to make sure I'm understanding. You have a native extension module that uses NumPy, and you want to distribute via distutils instead of setuptools? Something else?
Can pants support arbitrary subprojects which use setup.py rather than BUILD files?
It does support arbitrary subprojects with each being their own setuptools distribution. However, Pants auto-generates the setup.py file for you, e.g. to include only the
install_requires()
that is used by that subproject. You can still specify most the
setup()
kwargs, and you'd mostly copy and paste it from the
setup.py
file into the
BUILD
file. Check out https://www.pantsbuild.org/v2.4/docs/python-distributions. Is that similar to what you're looking for?
e
Thanks
You have a native extension module that uses NumPy, and you want to distribute via distutils instead of setuptools?
Sort of. I am using f2py via numpy.distutils to build the extension module like this
Copy code
from glob import glob
from numpy.distutils.core import Extension, setup


setup(
    ext_modules=[Extension(name="my_module", sources=glob("src/*.f90"))],
)
I am open to compiling the fortran code in another way, but pants didn’t have much support for non-python projects last time i recalled. Perhaps we should abstract the problem I bit. I have a PEP-517/518 compatible python project that I would like to manage in the same monorepo as the rest of my source code. For some reason or other, I don’t want to use pants for this subproject, but instead want to treat it as if it were a third-party dependency. If this subproject were in a separate repo and pushed to pypi, I could treat it as a third-party dependency. My question is if I can do this with a local source tree. Thanks
h
Interesting, I think I have an idea for this, more in a bit
Hmm, my idea didn't work, filed a ticket: https://github.com/pantsbuild/pants/issues/11942
This shouldn't be too hard to make work, since Pants uses pip under the covers for resolving requirements, but it looks like there are complications.
I take it back - it might be tricky to get invalidation right. Pants has to know what to cache the subproject against. Today when consuming 3rdparty dependencies it assumes (implicitly) that the requirement string (e.g., a local file:// URL in this case) is a valid cache key.
But in this case it wouldn't be - if you change something in that subproject, pants would have to know that it's no longer valid. But since it doesn't then know how to make it valid because it isn't building it...
So an alternative here would be to "publish" that subproject locally, against a unique version generated from hashes of the subproject's source files.
Pants itself could be made to handle that: it could run your external setup.py to build the project, assign it a version, and "publish" it to a local directory, and consume it from there.
To clarify, is this subproject built via setup.py? So there is a standard way of building it?
e
Sorry, I just your latest replies. The subproject is built via setup.py. Can you clarify what you mean by “standard way of building it”?
h
I mean, pants could just naively run
setup
on it? sounds like yes.
Basically I'm imagining a target that says "everything here is an external setup.py project, so just run setup on it"
And then consume it... somehow
need to figure out that last part
What would be the right way to consume this wheel without publishing it?
This seems basically doable