https://pantsbuild.org/ logo
Join Slack
Powered by
I’m trying to add a dependency (torch) which is av...
# general
f
I’m trying to add a dependency (torch) which is available from PyPI and the custom Pytorch domain. If I add the Pytorch domain to my 
pants.toml
with 
[python-repos].indexes.add
then run 
./pants generate-lockfiles
, the lockfile only has the package resolved from the custom domain, even though there are no wheels available for my platform from that domain, whereas wheels are available from PyPI for my domain. I need the extra index to be able to run my tests on CI. What can I do?
If I remove the 
[python-repos].indexes.add
, then pants correctly pulls the dependencies from PyPI
e
As a sanity check, what does `pants help-advanced python-repos`show for "current value:" ?
f
Copy code
current value: [
          "<https://pypi.org/simple/>",
          "<https://download.pytorch.org/whl/cpu>"
      ] (from pants.toml)
e
Looks good. Can you share the lockfile in full or is it sensitive?
f
Let me try to generate a minimal example
That might lend some insight, and then there won’t be any problems with sharing 🙂
e
Thank you! When I look at https://download.pytorch.org/whl/cpu it seems to be nowhere near a full index (no ansicolors, etc); so I'm not sure how your lock resolve could be fully satisfied from that index alone. It seems ~impossible not to have PyPI urls mixed into the lock and not get a lock failure outright up front.
f
Here’s where 
<http://downloads.pytorch.org|downloads.pytorch.org>
is disabled: https://github.com/bryanwweber/torch-lockfile-test/blob/e0f0492d95ee17be91f03cae2aefacda1e0b2c02/lock.txt
I’m on an M1, so I’d expect both repos to be listed among the choices
e
Aha, ok, both repos are listed! PyPI for typing extensions, the custom for PyTorch. So you're expecting the torch artifacts list to have a mix?
f
Yes, that’d be ideal so we can have multi-platform support in this lockfile
e
I think Pip is a blocker here - it doesn't work that way. Let me double-check.
f
OK, for this particular situation, I can resolve it by adding the extra index only for our CI system and generating the lockfiles there
e
It does look like Pip examines both sets of URLs and so Pex should be locking both sets. I'll experiment a bit more here and report back with either a Pex issue or an aha.
👍 1
Aha, I did not read closely enough. @future-oxygen-10553 your expectation is wrong! If you look in detail at the lock with the custom index added torch 
1.13.1+cpu
is locked. There are 0 artifacts for 
1.13.1+cpu
on PyPI, only for 
1.13.1
.
Basically, the pytorch folks are abusing a local build identifier (
+cpu
).
f
Yeah, I did notice that and hoped it wasn’t the case 😞
e
Yup, so Pex is doing the right thing here. And, it doesn't help you, but this is just one of 18 threads on problems with PyTorch. They just have not spent time integrating with the PyPA ecosystem it appears.
f
Yeah, I think it’s a positive step that they at least publish wheels now. For a while, it was really really hard to install with anything but Conda, IIUC
Anyway, my solution to this is to generate two lockfiles, and use 
pants.ci.toml
to specify the CPU one on CI
e
Aha, Ok. Well good. Maybe folks will nudge them further / contribute packaging fixes.
Anyway, my solution to this is to generate two lockfiles, and use 
pants.ci.toml
to specify the CPU one on CI
Yup, sounds right. I just wanted to make sure I did due diligence at squashing a Pex bug if there was one.
f
Thanks for following up on this!
4 Views
Open in Slack
PreviousNext
Powered by