tl;dr -

protobuf_sources

does not play nice with multiple resolves. @acceptable-football-32760 (or anyone else) - wonder if you've resolved this issue? I'm hitting the exact same roadblock :(

FWIW I use the approach that Alexey describes, but do so for all sources. Thankfully we don't have a lot of protos so I do it like this:

for resolve in ["cpu", "gpu", "base"]:
    protobuf_source(
        name=f"foobar_v1_{resolve}",
        source="foobar_v1.proto",
        grpc=True,
        python_resolve=resolve,
    )

    protobuf_source(
        name=f"foobar_v2_{resolve}",
        source="foobar_v2.proto",
        grpc=True,
        python_resolve=resolve,
        dependencies=[f":foobar_v1_{resolve}"],
    )


python_distribution(
    name="foobar_proto",
    dependencies=[
        ":foobar_v2_base",
    ],
    provides=python_artifact(
        name="foobar-proto",
        version="0.1.0",
        long_description_content_type="markdown",
    ),
)

Thank you for opening the issue! Agree about macros - I was about to implement it, when I realized the solution above (which works for me because i'm happy to live with one resolve at any point in time).

Hey @alert-psychiatrist-14102, for completeness: we ended up an extension of what I believe @bitter-ability-32190 suggested once (can't find to give credit) 1. we (writing our own

register.py

with a rule

torch_requirement

) added a flag

--torch-arch=[cpu|gpu]

which togather with the target

torch_requirement

chooses the necessary torch version 2. then we build lockfiles for resolves. When building lockfiles , we actually hack and always build into a default resolve which is

cpu

, but twice, using a script

# Generate lockfiles for `default`
./pants generate-lockfiles --resolve=python-default
# Move the generated lockfiles to `tmp`
mv tools/pants/lockfiles/python-default-cpu.lock ./tools/pants/lockfiles/python-default-cpu.lock.tmp
# Generate lockfiles for `gpu` -- "pretending" they are for `default`
./pants generate-lockfiles --torch-arch=gpu --resolve=python-default
# Move the generated lockfiles to be named `python-gpu`
mv tools/pants/lockfiles/python-default-cpu.lock ./tools/pants/lockfiles/python-gpu.lock
# Move the `tmp`, (which contains `default`) to `python-default-cpu`
mv tools/pants/lockfiles/python-default-cpu.lock.tmp ./tools/pants/lockfiles/python-default-cpu.lock

in

pants.toml

we have

[python.resolves]
# The command to generate both cpu and gpu lockfiles is a bit cumbersome.
# ❱ ./pants generate-lockfiles --resolve=python-default \
#  && mv tools/pants/lockfiles/python-default-cpu.lock ./tools/pants/lockfiles/python-default-cpu.lock.bak \
#  && ./pants generate-lockfiles --torch-arch=gpu --resolve=python-default \
#  && mv tools/pants/lockfiles/python-default-cpu.lock ./tools/pants/lockfiles/python-gpu.lock \
#  && mv tools/pants/lockfiles/python-default-cpu.lock.bak ./tools/pants/lockfiles/python-default-cpu.lock
python-default = "tools/pants/lockfiles/python-default-cpu.lock"
python-gpu = "tools/pants/lockfiles/python-gpu.lock"

And then, when building, we just provide different resolves as normal.

@acceptable-football-32760 - interesting! Thank you for sharing. This indeed sounds very similar. I'm curious how your

BUILD.pants

looks like, where you define the

python_requirements

targets. Which torch requirements do they see?