https://pantsbuild.org/ logo
#general
Title
# general
a

acceptable-football-32760

02/05/2022, 10:59 PM
Hello folks! Question about "target arch". The usecase is: we have dependency on
pytorch
which can be installed with CPU or GPU support. For tests, and having consistent local dev env, we would obviously need
CPU
. For deployment to training infra, we would need
GPU
. With my limited knowldge of πŸ‘– I see an imperfect solution: β€’ have tags on literally every target (because we have a lot of things depending on
pytorch
potentially transitively)
cpu
or
gpu
and duplicate those targets' code, changing only dependency on
torch_cpu
vs
torch_gpu
Question: is it possible, without messing tags, jsut somehow define a conditional
python_requirement
, say (in pseudocode)
Copy code
python_requirement(
    name = "torch_1.10",
    if (option == "cpu") {
        requirements = ["torch @ <https://download.pytorch.org/whl/cpu/torch-1.10.0%2Bcpu-cp38-cp38-linux_x86_64.whl>"]
    }
    else {
        requirements = ["torch @ <https://download.pytorch.org/whl/cu113/torch-1.10.0%2Bcu113-cp38-cp38-linux_x86_64.whl>"]
    }
)
maybe using options or smth like that. So that the choice is made at the level of the initial requirement. Something similar to Bazel's toolchain hosts... Any ideas are welcome! Thanks! Alex
πŸ‘€ 1
h

happy-kitchen-89482

02/06/2022, 4:32 PM
Hi @acceptable-football-32760 ! We're currently massively reworking how we support multiple conflicting requirements (AKA "multiple lockfiles". @hundreds-father-404 is the expert on this, so I will defer to them.
I'm digging a little deeper here, I don't see GPU-specific wheels on PyPI, so I guess they only exist at download.pytorch.org?
Oh, I see, the arch is technically part of the version (
1.10.0-cpu
vs.
1.10.0-cu113
)
So it sounds like in Pants 2.10.x (coming soon!) you'd want two named lockfiles,
cpu
and
gpu
, that differ only in that aspect (and those should be easy to create without a lot of repetition), and then your
python_sources
targets are set up to be compatible with both by default, but your tests always select
cpu
and your production deploy targets always select
gpu
. Something like that, again @hundreds-father-404 can be far more helpful than I can...
🀩 1
a

acceptable-football-32760

02/06/2022, 6:17 PM
Amazing! 🀩 Would it be possible to switch the requirement of prod targets to cpu (say for local development?) or the β€œtype” is fixed per-target? Ideally, I seek the possibility to switch the whole deployment to either of β€œarch”s. Similar situation would arise say in Bazel world when building beyween x64 and x86.
h

happy-kitchen-89482

02/07/2022, 1:30 AM
Good question, I will defer to Eric. Possibly you can select an "active lockfile" on the cmd line.
But I'm not sure what's planned
a

acceptable-football-32760

02/07/2022, 1:45 PM
Thank you @happy-kitchen-89482 πŸ‘ I'm looking forward to what @hundreds-father-404 tells! Maybe for such functionality, some kind of a custom plugin is possible
h

hundreds-father-404

02/07/2022, 4:46 PM
Hey Alexey, welcome! Indeed, 2.10 gained support for multiple "disjoint" resolves. https://pantsbuild.slack.com/archives/C046T6T9U/p1643922115275589?thread_ts=1643921413.397219&amp;cid=C046T6T9U We're working on a new feature that will make multiple overlapping resolves ergonomic, where
utils.py
can claim to work with both the resolve
pytorch-cpu
and
pytorch-gpu
In the meantime, I think this might work? 1. Define two resolves with
[python].resolves
,
pytorch-cpu
and
pytorch-gpu
. Set
compatible_resolves
on each
python_requirement
target to declare which requirement belongs in which resolve, including if both. 2. Change
[python].default_resolve
to the one you want. You won't need to update any targets for first-party code like
python_sources
, they'll use the default. 3. When you want to toggle to the other resolve, simply update
[python].default_resolve
. All your first party targets will use that one without needing to change any BUILD files. FYI you can set this option via env var and CLI arg, too: https://www.pantsbuild.org/docs/options How does that sound?
docs are coming soon - I've been updating our 2.10 docs to prepare for the upcoming release candidate
a

acceptable-football-32760

02/07/2022, 6:38 PM
Looks great @hundreds-father-404 (I already started implementing conditional dependency injection in 2.9.0 πŸ˜…) - does your solution with resovles work in 2.9.0 or we should wait till 2.10.0?
h

hundreds-father-404

02/07/2022, 6:42 PM
it is only part of 2.10.0 - I'll be releasing 2.10.0rc0 today and our focus this week will be stabilizing that into 2.10.0 https://www.pantsbuild.org/docs/release-strategy
I'll cut the release earlier today, rather than waiting to finish the docs rewrite (which might take another day). That way you can try this out even sooner πŸ™‚
Hey @acceptable-football-32760, the release is out. Set
pants_version = "2.10.0rc0"
You will need to set these options:
Copy code
[GLOBAL]
use_deprecated_python_macros = false

[python]
enable_resolves = true
Although before setting that, first run
./pants --version
for special instructions on how to fix the deprecation with
use_deprecated_python_macros
by using the
update-build-files
goal.
I'm still working on the docs today for "resolves", but https://www.pantsbuild.org/v2.10/docs/reference-python#section-resolves can get you started. I'm very happy to help with any questions also. This is one of the flagship new features of 2.10 (and one of our most requested features), so we're eager to get feedback on the feature πŸ™‚