acceptable-football-32760
08/15/2022, 12:50 PM2.12
]
[UPD2: I tried to parametrize the python_resolve
but then I get another error in š§µ ]
Hello dear community, I'm in the process of creating two resolves (a long-known problem of having gpu+cpu torch dependency) and decided to go via macros for python_sources
, python_requirements
and the likes -- macros which parametrize them like
def python_sources_custom(**kwargs):
kwargs.pop("resolve", None)
python_sources(resolve=parametrize("gpu", "cpu"), **kwargs)
All uneventful until I hit the grpc proto
generation and it seems that there's something off with them:
⢠on the one hand, protobuf_sources
target generator does not have anything like resolve
field (github) -- which makes sense provided that proto file is not bound to any specific language and resolve mechanism is python-specific -- so I can't actually parametrize them
⢠on the other hand, I hit an error for dependency resolver
UnownedDependencyError: Pants cannot infer owners for the following imports in the target some/path/to/file.py@resolve=gpu:
* some.import.of.this.file_pb2_grpc (line: 1)
These imports are not in the resolve used by the target (`gpu`), but they were present in other resolves:
* some.import.of.this.file_pb2_grpc: 'cpu' from some/import/of/this/file.proto
⢠a.k.a. "despite .proto
file itself being not subject to resolve, pants naturally wants theresulting pb2_grpc.py
file to be resolved"
It seems to be a dead-lock š
or am I missing something?python_resolves
, I get the following error
ValueError: Only fields which will be moved to generated targets may be parametrized, so target generator path/to/proto/target (with type protobuf_sources) cannot parametrize the {'python_resolve'} field.
python_resolve
fields are installed via register_plugin_field
and are not immediately moveable...***_custom
macros allowing me to automatically parametrize python_sources
and python_requirements
generators
def python_sources_custom(**kwargs):
kwargs.pop("resolve", None)
python_sources(resolve=parametrize("gpu", "cpu"), **kwargs)
⢠ā
For torch requirement itself, I created two versions of it
python_requirement(
name = "torch-cpu",
requirements = ["torch @ <https://download.pytorch.org/whl/cpu/torch-1.10.2%2Bcpu-cp39-cp39-linux_x86_64.whl>"],
resolve = "cpu",
)
# same for `gpu`
⢠ā
Minor hiccup happened with the python generated grpc stubs where I had to create two protobuf_sources
manually
protobuf_sources(
name = "protos-cpu",
grpc = True,
python_resolve = "cpu",
)
# same for `gpu`
⢠š here happens the problem: since some of my .proto
s include other protos, let's say main.proto
includes inc.proto
via proto3
s import "path/to/inc.proto"
directive, I started getting a warning:
15:17:12.27 [WARN] The target //path/to/main.proto:protos-cpu imports `path/to/inc.proto` in the file path/to/main.proto, but Pants cannot safely infer a dependency because more than one target owns this file, so it is ambiguous which to use: ['//path/to/inc.proto:protos-cpu', '//path/to/inc.proto:protos-gpu'].
In general, after many attempts to work out the "get torch of either cpu or gpu and make it a global switch" in various flavours (I happily lived with a custom generator for half a year until we introduced lockfiles to increase CI stability, now I made several attempts to solve it using multiple locks and resolves) it seems that the resolves mechanism does not accommodate this usecase well - solving each layer reveals more manual work on the next one and the problems pop up deeper.
I wonder if I'm doing something fundamentally wrong, because the task of having a) "all packages except one to be the same" and b) "switch that single one depending on some environment, e.g. dev/prod" -- should be relatively easy to accomplish, yet I fail miserably š
, meaning I likely don't wrap my mind around something simple here š
Please help, would be a pity to just force everything to use GPU torch even for CI tests! šhappy-kitchen-89482
08/16/2022, 2:43 AM