The <building distributions> docs states: ```Naive...
# plugins
d
The building distributions docs states:
Copy code
Naively, you might think that a python_distribution publishes all the code of all the python_source targets it transitively depends on
I really like how pants is smart when it builds wheels via
pants package
. However, I'm working on a plugin that requires me to build a wheel file that has all source from a set of source roots present in it. This wheel shouldn't point at any third party or first party dependencies. Is there a
Get
that I can write to convince pants to build this "uber-wheel" file for me?
h
If you're writing a plugin you can do whatever you want, even this abuse of the wheel format 🙂 I assume you're not planning to publish these wheels and they are purely a deployment format?
And that you've considered and rejected PEX, which is designed for this?
d
Correct on both accounts. I won't be publishing this wheel file. It's just for a one-time run. PEX almost fits my use case. However, I'll be running this artifact on a remote system that already has all of the 3rd party dependencies installed so bundling 3rd party dependencies and uploading the PEX file results in unnecessary bloat
Also, the "remote system" is Spark + Databricks and using PEX on Databricks requires making changes that potentially impact other jobs running on the cluster
But Databricks supports libraries scoped to a single execution context if those libraries are wheels
If I have a
PythonSourceFiles
, can I create a
DistBuildRequest
to build the wheel that I'm looking for?
h
[Shakes fist at Databricks for not supporting PEX]
👍 2
[Even though OSS Spark does]
You may not need any special behavior though: A python_distribution() target will build a wheel containing all the code it depends on, transitively, that is not owned be some other python_distribution() target.
So I guess my first question would be - will you have more than one of these things?
And will they overlap on some libraries?
d
A python_distribution() target will build a wheel containing all the code it depends on, transitively, that is not owned be some other python_distribution() target.
Yeah, I saw that on the "building distributions" page too and that's a really cool feature. Unfortunately, we are publishing multiple wheels. It's just that we also have a use case where it'd be great to just make an uber-wheel that holds our source files and doesn't reference any other 1st or 3rd party dependencies
h
Ah, so this will require a plugin, ya
d
Yep. So back to this question. Do you have any pointers on which internal pants things I could potentially re-use for this plugin? I have a
PythonSourceFiles
which has all of the target sources and source roots
h
This is all about setup.py generation, right? You could do this today if you wrote your own (simple in this case) setup.py
d
I'm not sure that I follow. The main goal is to get all of the sources into a single wheel. Does creating a simple setup.py by hand achieve that?
h
I think it would be a very simple one, yes
And Pants can work with a handwritten setup.py
It's in the docs, I'm too lazy at the moment to find the link 🙂
But I assume a handwritten setup.py that globs over everything you want it to would be very easy to write
d
Hi, thanks for your input on this. We decided to just go with the simplistic "push multiple wheels" approach as that was easier than trying to configure multiple wheel distributions for each project