Hello everyone! I have a bunch of git submodules in

3rdparty/

that have existing setup.py or pyproject.toml. Previously, we have installed them with

pip install -e --no-dependencies

into our environment. I'm trying to find a way to use them within the pants build system without making changes to the submodules such as adding pants

BUILD

files. I've tried a bunch of things, such as: 1. A

python_requirements(name="3rdparty")

with a

requirements.txt

listing all the deps, e.g.

-e 3rdparty/my_submodule

2. A manual list of

python_requirement

targets:

python_requirement(
    name="my_submodule",
    requirements=["my_submodule @ file:///absolute/path/3rdparty/my_submodule"],
)

3. Creating a

python_distribution()

target that points to the existing

setup.py

, e.g.

# 3rdparty/my_submodule/BUILD

files(name="files", sources=["setup.cfg", "README.md"])

python_sources()

python_distribution(
    name="my_submodule",
    dependencies=[
        ":files",
    ],
    provides=setup_py(
        name="my_submodule",
        setup_script="3rdparty/my_submodule/setup.py",
    ),
    generate_setup=False,
    # wheel=False,
)

Options 1 and 2 don't seem to work, and option 3 seems to require me to create BUILD files in all subdirectories of the submodule for the

python_distribution

to be correctly built. Is there a different option I'm missing? Is the

python_distribution()

the best approach, and is there a way to build the distribution without creating pants build files in the submodule?

Have you consider adding

3rdparty

in your

root_patterns

and let the dependency inference figure those out?

for this to work, would I have to add the correct submodule directories to the

root_patterns

, e.g.

3rdparty/my_submodule/src

instead of the general

3rdparty/

directory?

yes

If you can treat these as first-party sources (i.e. Joel's idea) (where

import my_submodule.foo

will specifically find

3rdparty/my_submodule/src/my_submodule/foo.py

and not need to install the whole submodule as a full distribution) that seems like a reasonable option, although if you have native code that needs to be build via

setup.py

(or other

setup.py

fanciness) this might not work. You'll also need to capture the dependencies of the submodules in your main repository resolve. For option 3, you could hypothetically use a single

3rdparty/BUILD

that handles all of the submodules, e.g.

files(name="my_submodule1-files, sources=["my_submodule/setup.cfg", ...])
python_sources(name="my_submodule1-src", sources=["my_submodule/**/*.py"])
python_distribution(
   name="my_submodule1"<
   dependencies=[":my_submodule1-files", ":my_submodule1-src"],
   ...
)

# same again for my_submodule2, ...

The common code could be factored out into a normal python function too, and/or written in a loop too:

def submodule_targets(name):
    file(name=f"{name}-files", sources=[f"{name}/setup.cfg", ...])
    ...

for submod in ["my_submodule1", "my_submodule2", ...]:
    submodule_targets(submod)

Thanks! I will experiment and see if all the submodules can be treated as first-party code. We work in ML, so it's popular to compile custom operations as part of the

setup.py

which might pose a problem

For 2, given that you’re not allowing to generate the setup.py file. I think you need to define the

python_requirement

target with information for

setuptools-scm

and then use that as a dependency for the distribution. Like you did for the files.

I tried something like this:

python_requirement(
    name="setuptools-scm-megapose",
    requirements=["setuptools-scm>=6.4.2"],
)

python_distribution(
    name="megapose",
    dependencies=[":megapose6d-files", ":megapose6d-src", ":setuptools-scm-megapose"],
    provides=python_artifact(name="megapose"),
    generate_setup=False,
    wheel=True,
)

is that the correct way to add the setuptools dep? What I typed above doesn't seem to fix the original

setuptools-scm

error

you got exactly the same error?

I get exactly the same error as I have before, but roughly similar to the one in the github issue:

❯ pants --keep-sandboxes=always test src/bdml/projects/megapose/tests:test_megapose_node
14:24:38.67 [INFO] Preserving local process execution dir /tmp/pants-sandbox-RXL852 for Building pytest_runner.pex
14:24:39.46 [INFO] Completed: Building pytest_runner.pex
14:24:39.46 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
Failed to resolve requirements from PEX environment @ /tmp/pants-sandbox-RXL852/local_dists.pex.
Needed cp310-cp310-manylinux_2_35_x86_64 compatible dependencies for:
 1: setuptools-scm>=6.4.2
    Required by:
      megapose 1.0
    But this pex had no ProjectName(raw='setuptools-scm', validated=False, normalized='setuptools-scm') distributions.

it tries to look for other python distributions in the project

I removed it from the

python_distribution

target and added it to the source target:

python_requirement(
    name="setuptools-scm-megapose",
    requirements=["setuptools-scm>=6.4.2"],
)

Unfortunately it yields the same error

have you tried to remove the

install_requires

as the issue that you mentioned suggest? Maybe there is a bug to be fixed here

removing the

install_requires

made the build error go away and the test ran!