In our project a few of the libraries depend on pytorch, so we have CPU and GPU versions of them, like:

python_source(
  name="lib",
  source="lib.py",
  dependencies=[
    ...
    "torch_cpu",
    ...
  ],
)

python_source(
  name="lib_gpu",
  source="lib.py",
  dependencies=[
    ...
    "torch_gpu",
    ...
  ],
)

When importing these libraries, we get a

[WARN] Pants cannot infer owners for the following imports in the target $TARGET_NAME

message, even though we clearly specify in the dependencies which one (ie,

lib

or

lib_gpu

) we are importing. Is there a way to fix this, other than suppressing the warning?

How are you defining

torch_gpu

and

torch_cpu

?

We have a global

requirements.txt

file, to lock the packages used by all libraries. We also have a

requirements_gpu.txt

for the GPU packages. Everything runs absolutely fine, so pants is able to get the correct packages. But it still gives the warning. Regarding you second question, no. We are locking all our packages directly in the requirements.txt files. What do we get by using resolves/lockfiles?

I'd suggest reading the docs; https://www.pantsbuild.org/docs/python-lockfiles#what-are-lockfiles. But in short it doesn't just pin your direct dependencies, but ensures all your transitive dependencies are also tracked. So it's a stronger guarantee, and automates a lot of dependency management woes you might have. Also reduces risk of supply-chain attacks since you'd have hashes for them.

Thanks! Regarding resolves: how do you handle different machines (Linux vs MacOS) requiring different versions of a package? Pytorch is again the offender here The requirements.txt file lets us have a conditional (

sys_platform == "linux"

), which is why we're doing it this way

Maybe just adding the exclusions makes the warning happy as it'd not even be allowed to consider the other ones. Fwiw what we have is three resolves: •

base

, which is

torch==1.12,!=1.12+cpu,!=1.12+cu116

•

gpu

, which is

torch==1.12+cu116

•

cpu

, which is

torch==1.12+cpu

Yes, all those constraints are necessary as python-repos config is global unfortunately.

Thanks for the help. I'll look into this 👍

Also. If you trust that it works (or have enough tests 😉); you should be able to use

# pants: no-infer-dep

on the import line. You can also set this globally with https://www.pantsbuild.org/docs/reference-python-infer#ignored_unowned_imports.

Hey @gorgeous-winter-99296: sorry to bother you again. I am looking into enabling locks but I'm running into some problems. Since you seem to have solved them, I have some questions for you. I have enabled locks and have 3 resolves: linux, mac, and gpu. They all have the same packages except for torch.

linux

is the default resolve. I then added a

resolve=parametrize("linux", "mac", "gpu")

option to every library importing torch. Everything works fine on linux (cpu). However, the other two resolves don't work:

NoCompatibleResolveException: The target path/to/library:lib@resolve=gpu uses the 'resolve' 'gpu', but some of its dependencies are not compatible with that resolve

It looks like I need to add the

resolve=parametrize(...)

line to every BUILD rule. Is there an alternative? Also, how do you select which version to run/test? Doing

pants run path/to/stuff:name@resolve=cpu

is very annoying. Plus we want to run all tests using the linux resolve for CI. While I might want to run all tests using the mac resolve on my laptop Can you provide some more details on how you dealt with this?

What we use is the

__defaults__

mechanism in a few select places in top-level BUILD-files:

__defaults__({
     python_sources: dict(resolve=parametrize(...)),
     python_tests: dict(resolve="cpu"),
})

This sets the resolve field for all "nested" targets from the view of that BUILD file. So pretty much every Python file that is used in our RL project has that resolve parametrization, but we only specify it in a few locations to apply recursively. Regarding the target selection when running etc, we have a

/cmd/

directory where we place entrypoint files to avoid specifying the full path. There we don't use

python_sources

, instead we use one `python_source`/`pex_binary` per file and set the resolve to whatever we want. We've played around with

--python-default-resolve

based workflows for those as well but it hasn't seen huge adoption so I'm unsure how well it works in practice.

Thanks! In terms of target selection it is a bit harder for us: some people work on linux while others work on mac, so we need to support those two 😕 I wish there was a simpler way to select the targets, like:

pants run path/to/target:bin --resolve=cpu

. And I also expected that if I didn't say anything it would default to the default resolve (i.e.,

pants run path/to/target:bin

should use the default resolve, rather than forcing me to pick with the

@resolve=name

thing). What is the

--python-default-resolve

flag?