First:

pybind11

1. I have a plugin which exposes a

http_file

target. It takes in URL+SHA+Length and downloads the file + a little bit of renaming for convenience

http_file(
    name="pybind11",
    url="<https://github.com/pybind/pybind11/archive/v2.4.3.tar.gz>",
    sha256="1eed57bc6863190e35637290f97a20c81cfe4d9090ac0a24f3bbf08f265eb71d",
    length=571127,
)

2. Specifically, the

http_file

target is a codegenerator generating a

FileSourceField

so that it gets consumed like a file. Now I can use a dependency on

//3rdparty/deps:pybind11

and expect the file in the sandbox (relevant issue: https://github.com/pantsbuild/pants/issues/13589)

Second: Py C Extension Module (this one is tricky) 1. Since this gets consumed like any other Python source, it is defined as a PythonSourceField generator 2. Make a target for

py_extension_module

and use it, providing it the relevant

setup.py

,

//3rdparty/deps:pybind11

and Cpp file(s) 3. Then the generator... It's a cobbled together collection of snippets from

pants/backend/python/goals/setup_py.py

and

pants/backend/python/util_rules/python_sources.py

a. We can't request codegen Python sources (possibly needed by

setup.py

, because then we'll hit a circular loop. Womp womp) b. We build a request to build the Wheel only out of the specified deps c. Rename the wheel to

.zip

and ask Pants to extract the contents d. Extract the

.so

from the unzipped archive e. Return the

GeneratedSources

with the correct directory prefix

Last: Dependency inference This one is straightforward. implement an

InferDependenciesRequest

by mapping imports to C Extension Module targets

Oh and my

setup.py

looks like:

import tarfile

try:
    from setuptools import Extension, setup
except ImportError:
    from distutils.core import Extension, setup

# @TODO: Only extract the relevant dir. `extract` didn't work
tarfile.open(name="../../../../3rdparty/deps/pybind11", mode="r:gz").extractall()


setup(
    name="my_name",
    ext_modules=[
        Extension(
            name="my_name",
            include_dirs=["pybind11-2.4.3/include"],
            sources=["my_name.cpp"],
        ),
    ],
)

Eventually it'll be cool to upstream the changes, but with as many hacks/assumptions I baked in it'll be a long road

Correction, if I make my Py Extension Module a File generator I don't have a circular dep issue 🎉

(Also this gets us closer to a

mypyc

in-repo files solution)