https://pantsbuild.org/ logo
a

acceptable-football-32760

08/15/2022, 12:50 PM
[UPD: ver ==
2.12
] [UPD2: I tried to parametrize the
python_resolve
but then I get another error in šŸ§µ ] Hello dear community, I'm in the process of creating two resolves (a long-known problem of having gpu+cpu torch dependency) and decided to go via macros for
python_sources
,
python_requirements
and the likes -- macros which parametrize them like
Copy code
def python_sources_custom(**kwargs):
    kwargs.pop("resolve", None)

    python_sources(resolve=parametrize("gpu", "cpu"), **kwargs)
All uneventful until I hit the grpc
proto
generation and it seems that there's something off with them: ā€¢ on the one hand,
protobuf_sources
target generator does not have anything like
resolve
field (github) -- which makes sense provided that proto file is not bound to any specific language and resolve mechanism is python-specific -- so I can't actually parametrize them ā€¢ on the other hand, I hit an error for dependency resolver
Copy code
UnownedDependencyError: Pants cannot infer owners for the following imports in the target some/path/to/file.py@resolve=gpu:

  * some.import.of.this.file_pb2_grpc (line: 1)

These imports are not in the resolve used by the target (`gpu`), but they were present in other resolves:

  * some.import.of.this.file_pb2_grpc: 'cpu' from some/import/of/this/file.proto
ā€¢ a.k.a. "despite
.proto
file itself being not subject to resolve, pants naturally wants theresulting
pb2_grpc.py
file to be resolved" It seems to be a dead-lock šŸ˜… or am I missing something?
IF I parametrize
python_resolves
, I get the following error
Copy code
ValueError: Only fields which will be moved to generated targets may be parametrized, so target generator path/to/proto/target (with type protobuf_sources) cannot parametrize the {'python_resolve'} field.
Indeed seems that
python_resolve
fields are installed via
register_plugin_field
and are not immediately moveable...
I think, after going down the path of multiple resolves for introducing cpu/gpu target arch for Torch, I eventually hit a roadblock. So what I did: ā€¢ āœ… I created
***_custom
macros allowing me to automatically parametrize
python_sources
and
python_requirements
generators
Copy code
def python_sources_custom(**kwargs):
    kwargs.pop("resolve", None)

    python_sources(resolve=parametrize("gpu", "cpu"), **kwargs)
ā€¢ āœ… For torch requirement itself, I created two versions of it
Copy code
python_requirement(
    name = "torch-cpu",
    requirements = ["torch @ <https://download.pytorch.org/whl/cpu/torch-1.10.2%2Bcpu-cp39-cp39-linux_x86_64.whl>"],
    resolve = "cpu",
)

# same for `gpu`
ā€¢ āœ… Minor hiccup happened with the python generated grpc stubs where I had to create two
protobuf_sources
manually
Copy code
protobuf_sources(
    name = "protos-cpu",
    grpc = True,
    python_resolve = "cpu",
)

# same for `gpu`
ā€¢ šŸ›‘ here happens the problem: since some of my
.proto
s include other protos, let's say
main.proto
includes
inc.proto
via
proto3
s
import "path/to/inc.proto"
directive, I started getting a warning:
Copy code
15:17:12.27 [WARN] The target //path/to/main.proto:protos-cpu imports `path/to/inc.proto` in the file path/to/main.proto, but Pants cannot safely infer a dependency because more than one target owns this file, so it is ambiguous which to use: ['//path/to/inc.proto:protos-cpu', '//path/to/inc.proto:protos-gpu'].
In general, after many attempts to work out the "get torch of either cpu or gpu and make it a global switch" in various flavours (I happily lived with a custom generator for half a year until we introduced lockfiles to increase CI stability, now I made several attempts to solve it using multiple locks and resolves) it seems that the resolves mechanism does not accommodate this usecase well - solving each layer reveals more manual work on the next one and the problems pop up deeper. I wonder if I'm doing something fundamentally wrong, because the task of having a) "all packages except one to be the same" and b) "switch that single one depending on some environment, e.g. dev/prod" -- should be relatively easy to accomplish, yet I fail miserably šŸ˜…, meaning I likely don't wrap my mind around something simple here šŸ™ˆ Please help, would be a pity to just force everything to use GPU torch even for CI tests! šŸ™
h

happy-kitchen-89482

08/16/2022, 2:43 AM
@hundreds-father-404, can you weigh in here? Thanks!
šŸ™ 1
5 Views