Hey, continuing off from the conversation in #general, I'm looking to primarily build Cython code (

python setup.py build_ext --inplace

), though there are tests associated and running those is a secondary goal, which are in Python

Cool, thanks. I’m not super familiar with Cython - to check my understanding, you’re running setup.py to convert a .pyx file into an .so file, and then checking in that .so file so that your normal Python files can use it? Are you also doing thing like directly running tests on the .pyx files themselves? This sounds like codegen might fit your use case. Whenever normal Python code depends on your Cython code, Pants will run your custom build gen implementation to create the .so files

Correct, the general goal is to turn .pyx and .h files into a .so file. My current WIP code is based on the protobuf example in the codegen section of the documentation. It wasn't clear to me that I should use another

Sources

subclass to represent the .so file output, as strictly speaking it's a binary, not a source file. Testing is mixed Python and C++ code. We're only really interested in the Python code at this time, which should be easy to create the dependency once the

Target

is figured out

Okay, cool. So I think you will want to use codegen. Check out a guide at https://www.pantsbuild.org/docs/plugins-codegen I think you will likely want to follow this roadmap: 1. Create a custom target type called

cython_library

. Use a subclass of

Sources

called

CythonSources

, and set

expected_file_extensions = (".h", ".pyx")

. This subclass is important for codegen to work so that we can say what the “input” type is. 2. Subclass

GenerateSourcesRequest

, and set

input = CythonSources

and

output = ResourcesSources

. I don’t think

ouptut = PythonSources

would accurately reflect what’s going on. You’ll want to use resources because call sites need to say what types of sources they’re expecting; right now, the builtin Python backend says it works with

PythonSources

,

FilesSources

, and

ResourcesSources

(or subclasses). If you were generating something else like

JavaSources

, then the Python backend would ignore your code gen implementation. 3. Your rule will take as input the

GenerateSourcesRequest

subclass and will return

GeneratedSources

. 4. To run setuptools, you can use the

setuptools.pex

that we have in

pants.backend.python.goals.setup_py

. Specifically, in your rule, request the type

SetuptoolsSetup

. This will give you back a Pex with

setuptools

in it, and the entry-point set to run setuptools. Check out that file for some examples of how it’s used, and we’re happy to help

How would I "override" the existing setup_setuptools rule?

You can create a new version of the rule with the config you’d like. More or less, copy and paste the original rule, and have it return a new, distinct type. The distinct type is to avoid ambiguity in the rule graph, where you would have multiple ways of the engine computing the same

SetuptoolsSetup

type. (Or, you can inline the logic into your codegen rule. No need for a dedicated rule and

@dataclass

to set up

setuptools/cython

. The factored out rule is only to faciliate reuse, but if you’re only using this in one place, it’s simpler to inline.)