https://pantsbuild.org/ logo
#general
Title
# general
f

future-oxygen-10553

02/13/2023, 3:42 PM
I’m trying to add a dependency (torch) which is available from PyPI and the custom Pytorch domain. If I add the Pytorch domain to my
pants.toml
with
[python-repos].indexes.add
then run
./pants generate-lockfiles
, the lockfile only has the package resolved from the custom domain, even though there are no wheels available for my platform from that domain, whereas wheels are available from PyPI for my domain. I need the extra index to be able to run my tests on CI. What can I do?
If I remove the
[python-repos].indexes.add
, then pants correctly pulls the dependencies from PyPI
e

enough-analyst-54434

02/13/2023, 3:52 PM
As a sanity check, what does `pants help-advanced python-repos`show for "current value:" ?
f

future-oxygen-10553

02/13/2023, 3:57 PM
Copy code
current value: [
          "<https://pypi.org/simple/>",
          "<https://download.pytorch.org/whl/cpu>"
      ] (from pants.toml)
e

enough-analyst-54434

02/13/2023, 4:02 PM
Looks good. Can you share the lockfile in full or is it sensitive?
f

future-oxygen-10553

02/13/2023, 4:03 PM
Let me try to generate a minimal example
That might lend some insight, and then there won’t be any problems with sharing 🙂
e

enough-analyst-54434

02/13/2023, 4:06 PM
Thank you! When I look at https://download.pytorch.org/whl/cpu it seems to be nowhere near a full index (no ansicolors, etc); so I'm not sure how your lock resolve could be fully satisfied from that index alone. It seems ~impossible not to have PyPI urls mixed into the lock and not get a lock failure outright up front.
f

future-oxygen-10553

02/13/2023, 4:15 PM
Here’s where
<http://downloads.pytorch.org|downloads.pytorch.org>
is disabled: https://github.com/bryanwweber/torch-lockfile-test/blob/e0f0492d95ee17be91f03cae2aefacda1e0b2c02/lock.txt
I’m on an M1, so I’d expect both repos to be listed among the choices
e

enough-analyst-54434

02/13/2023, 4:17 PM
Aha, ok, both repos are listed! PyPI for typing extensions, the custom for PyTorch. So you're expecting the torch artifacts list to have a mix?
f

future-oxygen-10553

02/13/2023, 4:18 PM
Yes, that’d be ideal so we can have multi-platform support in this lockfile
e

enough-analyst-54434

02/13/2023, 4:18 PM
I think Pip is a blocker here - it doesn't work that way. Let me double-check.
f

future-oxygen-10553

02/13/2023, 4:26 PM
OK, for this particular situation, I can resolve it by adding the extra index only for our CI system and generating the lockfiles there
e

enough-analyst-54434

02/13/2023, 4:33 PM
It does look like Pip examines both sets of URLs and so Pex should be locking both sets. I'll experiment a bit more here and report back with either a Pex issue or an aha.
👍 1
Aha, I did not read closely enough. @future-oxygen-10553 your expectation is wrong! If you look in detail at the lock with the custom index added torch
1.13.1+cpu
is locked. There are 0 artifacts for
1.13.1+cpu
on PyPI, only for
1.13.1
.
Basically, the pytorch folks are abusing a local build identifier (
+cpu
).
f

future-oxygen-10553

02/15/2023, 4:37 PM
Yeah, I did notice that and hoped it wasn’t the case 😞
e

enough-analyst-54434

02/15/2023, 4:38 PM
Yup, so Pex is doing the right thing here. And, it doesn't help you, but this is just one of 18 threads on problems with PyTorch. They just have not spent time integrating with the PyPA ecosystem it appears.
f

future-oxygen-10553

02/15/2023, 4:38 PM
Yeah, I think it’s a positive step that they at least publish wheels now. For a while, it was really really hard to install with anything but Conda, IIUC
Anyway, my solution to this is to generate two lockfiles, and use
pants.ci.toml
to specify the CPU one on CI
e

enough-analyst-54434

02/15/2023, 4:39 PM
Aha, Ok. Well good. Maybe folks will nudge them further / contribute packaging fixes.
Anyway, my solution to this is to generate two lockfiles, and use
pants.ci.toml
to specify the CPU one on CI
Yup, sounds right. I just wanted to make sure I did due diligence at squashing a Pex bug if there was one.
f

future-oxygen-10553

02/15/2023, 4:40 PM
Thanks for following up on this!
4 Views