Can I use different set of custom PIP indexes for different resolves? Pants/Pex doesn't generate a portable lockfile if we include an index in our primary lockfile; but we need it for other more niche applications and workflows.

Oh hey we do something similar

Hmm, but how do you handle local work? I.e., some users on Mac, some users on Linux needing cuda support?

I don't. CUDA support is enabled by switching the default resolve to the CUDA one. We also dont' support Mac in our repo You can coerce Pex to lock for both Linux/Mac, and use that as the default.

Ah, yeah. Mac support is the crunch here. Locking with anything with a local tag makes the lockfile not work on Mac. And just having those local tags in an index makes them get picked.

So it sounds like you have 2 problems then? Linx/Mac and CPU/GPU?

They interact. If I only care about Linux I can have local tags in two different resolves and it works. But that prevents it from working on Mac, ever. Just having the index is enough to break that. And then if I use the PyPi it works on Linux/Mac, but then the torch version ends up not supporting the cards we use for training locally (3090/sm86).

Wait, then yeah you can follow the steps above. The Linux/Mac resolve uses PyPI. The Linux+GPU resolve uses PyPI+Torch index

Hmm, maybe I'm misunderstanding it. Are your steps just instead of

pants generate-lockfiles

?

Yup!

I wonder if this would work...

pants generate-lockfiles --resolve=cpu
pants generate-lockfiles --python-indexes='["<http://pypi.org/simple|pypi.org/simple>", "<http://download.torch.org/cu117|download.torch.org/cu117>"]' --resolve=gpu

pants generate-lockfiles

tries to do a

universal

lock 🤮

And having those being the same reqs file but two different

torch==...

statements

Also if a package gets updated on PyPI in between those calls, your versions might differ

It's only important from a "sanity" standpoint, right? As in, I could potentially be running different code for CPU and GPU, but in a good scenario it doesn't matter if cloudpickle 0.13.1 vs 0.13.2 is used. But a pain to debug if there is a minor regression.

Yeah. I can't in good conscience let them float for my org. Either way I think you'll get bit by the universal lockfiles for the second command. Try it though and see what happens

I use one already, the other requirements I add via build file. 😛

@gorgeous-winter-99296, hey, you could take a look how I handle it: https://github.com/pantsbuild/pants/issues/18293#issuecomment-1439217661

Yeah; I've considered something like that. Just a pain to manage private mirrors... one more step for onboarding, etc. Permissions. Maintenance.

IIRC the python spec says that local tags are just sugar and can be ignored. Or something to that effect

Yepp, but also:

Local versions sort differently, this PEP requires that they sort as greater than the same version without a local

(from https://peps.python.org/pep-0440/)

@gorgeous-winter-99296 to your question, if there's a bug there it's in Pex not refusing to lock. A fundamental constraint in any style lock that Pex generates is it must pick exactly 1 version for each project in the lock. This is a limitation of the lock mechanism, which is delegation to Pip: it's just trying to pick a single version like it always does.

FWICT torch should be using extras, say [CPU] or [GPU] to accomplish this. That would require they change how they package a bit and it's likely a sailed ship.

Yeah that's my hot take as well. I forgot to mention it to the Metas folks I met at PyCon 😕

Yeah. It's not like we have much leverage as external users. It's not like tensorflow has much better packaging...

mxnet

uses different package names. So that's a pro here, but then there's other cons 😛

@enough-analyst-54434 to expand a bit on "is there a bug here", I specify a requirement like

torch==1.11.0

and repos like:

[python-repos]
indexes = [
        "<https://pypi.org/simple/>",
        "<https://download.pytorch.org/whl/cpu/>"
]

and do a lock via pants and get

{
  "artifacts": [
    {
      "algorithm": "sha256",
      "hash": "544c13ef120531ec2f28a3c858c06e600d514a6dfe09b4dd6fd0262088dd2fa3",
      "url": "<https://download.pytorch.org/whl/cpu/torch-1.11.0%2Bcpu-cp39-cp39-linux_x86_64.whl>"
    }
  ],
  "project_name": "torch",
  "requires_dists": [
    "typing-extensions"
  ],
  "requires_python": ">=3.7.0",
  "version": "1.11.0+cpu"
}

Which seems... like the wrong thing to do. So it's locking a different, actually incompatible version than the one specified. If it was truly universal there I'd expect it to fallback to the pypi version, which would work for both.

Ah yeah, clever!

@gorgeous-winter-99296 what’s next with these resolves, how do you use them? For example, I have a resolver data-science for ml specific and it has inside

torch

. How do you accomplish to use right resolver in CI and right on mac/linux users?

That's tomorrows problem. With lots of pain, I imagine. I'll report back!

Thank you very much, if it works for you, then I think it would be great to see your report in that GitHub issue

@gorgeous-winter-99296 re torch==1.11.0 selecting 1.11.0+X, this is just Pip following the rules: https://peps.python.org/pep-0440/#version-matching towards the bottom you'll find: "If the specified version identifier is a public version identifier (no local version label), then the local version label of any candidate versions MUST be ignored when matching versions."

But isn't the requirement of a universal lock that it works on all targeted platforms, i.e. Linux and Mac when using Pants? So the choice isn't valid.

Ah, I was just focused on the version not the artifact. I'm AFK through the 8th but can try to repro your example and discover more. What Pip version are you using? Basically if you could provide the lock from one of the 3 successful examples above, that would help gather relevant details in 1 place.

I'll see if I can split this out in a separate repository!

Ok, I think the issue here is my mis-portrayal of

--target-system

. That just limits what artifacts Pex locks. I.E: It does not try to lock Windows artifacts. It locks any Linux and Mac artifacts available though. This does not mean it somehow tests the locks will work on Linux and Mac. In fact, universal locks are conceptually broken this way in general. The "universal" claim is best effort. If your universe includes, say, Mac arm and there are no wheels released for Mac arm for some set of your transitive requirements, the end result is yolo. Who knows, some of those sdists may never build on Mac arm and the lock never work for those. So, if a lock grabs some pin - all this torch shenanigans with local versions aside - and that pin has just a - say - Linux x84_64 wheel available with no sdist, the universal lock will succeed.

Right. So it's a maximum set; not a minimum set. That makes sense, as otherwise adding

+cpu

wouldn't be lockable instead. If I understand correctly, adding

--platform

or the complete dito would still only be validation, right? It wouldn't actually constrain the resolve process. I know from attempting to work around this issue in both PDM and Poetry that it's equally borked everywhere; and the hacks are equally ugly. If I understand you correctly; if Pants would support a multi-platform lock this would be solvable (albeit maybe quite complex). Is there a philosophical reason Pants does not support this, or just no-one having been hurt enough by it to fix it? It doesn't seem like an insurmountable contribution to make if it'd be accepted. OTOH; I now have a documented workable approach that'll continue working for the foreseeable future. I expect though that when we upgrade torch to 1.13 or 2.0 when it starts pulling in CUDA we'll have another fun set of problems to solve... Which might necessitate the multi-platform lock anyways.

I'm not really sure. I just did the Pex work to make locking possible. Others added Pants support and I can't represent their logic. You should definitely ask around more widely.

I'm almost certain it's possible to do. Feel free to open an issue

There's this issue already; https://github.com/pantsbuild/pants/issues/18293

FWIW @gorgeous-winter-99296 a

===

may work for your non-local-version resolve instead of having to add the

!=

clauses. But sticking to what you came up with may be your best bet. I think either way takes a book to explain.

I agree @gorgeous-winter-99296, related but not the same. I think this more dedicated issue could address some of the larger one

👍 Sounds good, will try to do a write-up.