https://pantsbuild.org/ logo
Join Slack
Powered by
I'm trying to set up multiple resolves where the o...
# general
p
I'm trying to set up multiple resolves where the only "real" difference is the set of 
--extra-index-urls
each uses (because pytorch has CUDA and non-CUDA libs which are differentiated only by the index url). But it appears that 
python-repos.indexes
is a global property and can't be set per-resolve. Is there a way to do this? (see https://pantsbuild.slack.com/archives/C046T6T9U/p1664233267501069 for some more context if necessary)
@bitter-ability-32190 you seemed to have an idea for how to do this on that previous thread...
b
Isn't error -9 sigkill?
p
doh! Sorry @bitter-ability-32190 referenced the wrong previous discussion. Will update the post here and tag the right person.
oh! you are the right person @bitter-ability-32190 but the relevant prior discussion is this one: https://pantsbuild.slack.com/archives/C046T6T9U/p1664233267501069
I think maybe I could do it by specifying the requirements by URL instead of 
indexes
but that seems brittle.
h
Nothing (environment marker, extras) distinguishes those wheels other than the index they’re fetched from?
p
correct.
At least that's how I understand it. Here's the docs: https://pytorch.org/get-started/locally/
There's a little interactive matrix where you select install methods (e.g. pip), versions of things, etc. and it spits out a 
pip install
command to run. As far as I can tell the only thing that changes is the 
--extra-index-url
. Weird way to do things IMHO...
@happy-kitchen-89482 @bitter-ability-32190 if y'all had any advice here it'd be much appreciated.
b
I've got nothing off the top of my head 😕 Might require some changes to Pants.
Except I think the extra index url might be for lockfile generation. Then on the consumption side you just have each lockfile? PEX lockfiles store the url I think
p
Yes, the full URLs are in the lockfile. Pants build the lockfiles for us via 
*generate*-lockfiles
and that gets re-run all the time as we add dependencies. And that goal uses the one-true 
python-repos.indexes
to populate the lockfiles even if you have mutliple resolves. I'm open to generating our own lockfiles manually somehow but unclear on how to do that. The pants lockfiles do not look like the output of 
pip freeze
...
h
I guess one thing you can do is change python-repos.indexes via env var when you generate each lockfile
So your “manual lockfile generation” is just “run Pants with a different setting for that option”
via flag or env var
p
Oh! I forgot we can manipulate those options via flags/vars. That works nicely. Indeed we have a 
doit
(https://pydoit.org/) wrapper around Pants so it can coordinate things like building containers that contain code build from multiple different languages, etc. so we could easily add it there and "hide the ugly" from users. We probably don't even need multiple resolves then as the normal day-to-day use-case is the CPU version and then we do a single build for production with GPU enabled. Thanks!!
5 Views
Open in Slack
PreviousNext
Powered by