#general
Title
f
future-oxygen-10553
02/13/2023, 3:42 PMI’m trying to add a dependency (torch) which is available from PyPI and the custom Pytorch domain. If I add the Pytorch domain to my
pants.toml[python-repos].indexes.add./pants generate-lockfilesIf I remove the
[python-repos].indexes.adde
enough-analyst-54434
02/13/2023, 3:52 PMAs a sanity check, what does `pants help-advanced python-repos`show for "current value:" ?
f
future-oxygen-10553
02/13/2023, 3:57 PM Copy code
current value: [
"<https://pypi.org/simple/>",
"<https://download.pytorch.org/whl/cpu>"
] (from pants.toml)e
enough-analyst-54434
02/13/2023, 4:02 PMLooks good. Can you share the lockfile in full or is it sensitive?
f
future-oxygen-10553
02/13/2023, 4:03 PMLet me try to generate a minimal example
That might lend some insight, and then there won’t be any problems with sharing 🙂
e
enough-analyst-54434
02/13/2023, 4:06 PMThank you! When I look at https://download.pytorch.org/whl/cpu it seems to be nowhere near a full index (no ansicolors, etc); so I'm not sure how your lock resolve could be fully satisfied from that index alone. It seems ~impossible not to have PyPI urls mixed into the lock and not get a lock failure outright up front.
f
future-oxygen-10553
02/13/2023, 4:15 PMHere’s where
<http://downloads.pytorch.org|downloads.pytorch.org>And here’s with it enabled: https://github.com/bryanwweber/torch-lockfile-test/blob/5b7e928a9bf4becbdbfe136d47c0e6a4503c687d/lock.txt
I’m on an M1, so I’d expect both repos to be listed among the choices
e
enough-analyst-54434
02/13/2023, 4:17 PMAha, ok, both repos are listed! PyPI for typing extensions, the custom for PyTorch. So you're expecting the torch artifacts list to have a mix?
f
future-oxygen-10553
02/13/2023, 4:18 PMYes, that’d be ideal so we can have multi-platform support in this lockfile
e
enough-analyst-54434
02/13/2023, 4:18 PMI think Pip is a blocker here - it doesn't work that way. Let me double-check.
f
future-oxygen-10553
02/13/2023, 4:26 PMOK, for this particular situation, I can resolve it by adding the extra index only for our CI system and generating the lockfiles there
e
enough-analyst-54434
02/13/2023, 4:33 PMIt does look like Pip examines both sets of URLs and so Pex should be locking both sets. I'll experiment a bit more here and report back with either a Pex issue or an aha.
👍 1
Aha, I did not read closely enough. @future-oxygen-10553 your expectation is wrong! If you look in detail at the lock with the custom index added torch
1.13.1+cpu1.13.1+cpu1.13.1Basically, the pytorch folks are abusing a local build identifier (
+cpuf
future-oxygen-10553
02/15/2023, 4:37 PMYeah, I did notice that and hoped it wasn’t the case 😞
e
enough-analyst-54434
02/15/2023, 4:38 PMYup, so Pex is doing the right thing here. And, it doesn't help you, but this is just one of 18 threads on problems with PyTorch. They just have not spent time integrating with the PyPA ecosystem it appears.
f
future-oxygen-10553
02/15/2023, 4:38 PMYeah, I think it’s a positive step that they at least publish wheels now. For a while, it was really really hard to install with anything but Conda, IIUC
Anyway, my solution to this is to generate two lockfiles, and use
pants.ci.tomle
enough-analyst-54434
02/15/2023, 4:39 PMAha, Ok. Well good. Maybe folks will nudge them further / contribute packaging fixes.
Anyway, my solution to this is to generate two lockfiles, and useYup, sounds right. I just wanted to make sure I did due diligence at squashing a Pex bug if there was one.to specify the CPU one on CIpants.ci.toml
f
future-oxygen-10553
02/15/2023, 4:40 PMThanks for following up on this!
4 Views