straight-action-80318
07/05/2022, 3:05 PM[python-repos]
indexes.add = ["<https://download.pytorch.org/whl/cu116>"]
in my pants.toml
file, but this makes my generate-lockfiles
time much much longer (probably 2-3x). I can’t figure out if there’s a way to only include this repo for the single pytorch
requirement.hundreds-father-404
07/05/2022, 3:08 PMI can’t figure out if there’s a way to only include this repo for the single pytorch requirement.There is not
but this makes my generate-lockfiles time much much longer (probably 2-3x)Hm, is it using a prebuilt wheel, or now you're building the wheel from an sdist? If the latter, that would explain the slowdown
bitter-ability-32190
07/05/2022, 3:10 PMpytorch
is HUGE FWIW, so I'm not terribly surprised.
The crux of the issue lies in Python packaging. In Python there is no way to get the metadata for a package without downloading the entire package. For packages like pytorch
that forces lockfile generation to download the entire thing just to parse a simple textfile 😞straight-action-80318
07/05/2022, 3:14 PM--extra-index-url <https://download.pytorch.org/whl/cu116>
to the requirements file?-i
flag but it doesn’t seem to work for Pantshundreds-father-404
07/05/2022, 3:17 PM-i
for the whole lockfile generation process.
But I'm not convinced adding the extra index is specifically slowing down resolution of non-Pytorch requirements. To test that, you could have a simple [python].resolves
w/ only requirements like ansicolors
and requests
in the resolve. Leave out Pytorch. Then time how long generating that lockfile takes w/ the index and w/o.straight-action-80318
07/05/2022, 3:42 PMhundreds-father-404
07/05/2022, 3:44 PM-ldebug generate-lockfiles
to see the argv that Pants uses when running Pex. Or recreate it yourself with something like pex3 -vvv lock create pytorch -i ...
bitter-ability-32190
07/05/2022, 3:45 PMstraight-action-80318
07/05/2022, 3:52 PMenough-analyst-54434
07/05/2022, 4:00 PMis there no way to passFWIW, even if Pants allowed you to do this, it wouldn't help - Pip uses the extra index globally and not just for the requirement it is next to in a requirements file. See: https://pip.pypa.io/en/stable/reference/requirements-file-format/#global-optionsto the requirements file?--extra-index-url <https://download.pytorch.org/whl/cu116>
straight-action-80318
07/05/2022, 4:01 PMfile:///path/to/index
bitter-ability-32190
07/05/2022, 4:02 PMenough-analyst-54434
07/05/2022, 4:03 PMstraight-action-80318
07/05/2022, 4:04 PMenough-analyst-54434
07/05/2022, 4:06 PM; ...
environment marker on the end to make sure each wheel is only downloaded for the appropriate python versions and platform. I'm not sure we've had anyone use that trick.straight-action-80318
07/05/2022, 4:14 PM