straight-action-8031807/05/2022, 3:05 PM
[python-repos] indexes.add = ["<https://download.pytorch.org/whl/cu116>"]
file, but this makes my
time much much longer (probably 2-3x). I can’t figure out if there’s a way to only include this repo for the single
hundreds-father-40407/05/2022, 3:08 PM
I can’t figure out if there’s a way to only include this repo for the single pytorch requirement.There is not
but this makes my generate-lockfiles time much much longer (probably 2-3x)Hm, is it using a prebuilt wheel, or now you're building the wheel from an sdist? If the latter, that would explain the slowdown
bitter-ability-3219007/05/2022, 3:10 PM
is HUGE FWIW, so I'm not terribly surprised. The crux of the issue lies in Python packaging. In Python there is no way to get the metadata for a package without downloading the entire package. For packages like
that forces lockfile generation to download the entire thing just to parse a simple textfile 😞
straight-action-8031807/05/2022, 3:14 PM
to the requirements file?
flag but it doesn’t seem to work for Pants
hundreds-father-40407/05/2022, 3:17 PM
for the whole lockfile generation process. But I'm not convinced adding the extra index is specifically slowing down resolution of non-Pytorch requirements. To test that, you could have a simple
w/ only requirements like
in the resolve. Leave out Pytorch. Then time how long generating that lockfile takes w/ the index and w/o.
straight-action-8031807/05/2022, 3:42 PM
hundreds-father-40407/05/2022, 3:44 PM
to see the argv that Pants uses when running Pex. Or recreate it yourself with something like
pex3 -vvv lock create pytorch -i ...
bitter-ability-3219007/05/2022, 3:45 PM
straight-action-8031807/05/2022, 3:52 PM
enough-analyst-5443407/05/2022, 4:00 PM
is there no way to passFWIW, even if Pants allowed you to do this, it wouldn't help - Pip uses the extra index globally and not just for the requirement it is next to in a requirements file. See: https://pip.pypa.io/en/stable/reference/requirements-file-format/#global-optionsto the requirements file?
straight-action-8031807/05/2022, 4:01 PM
bitter-ability-3219007/05/2022, 4:02 PM
enough-analyst-5443407/05/2022, 4:03 PM
straight-action-8031807/05/2022, 4:04 PM
enough-analyst-5443407/05/2022, 4:06 PM
environment marker on the end to make sure each wheel is only downloaded for the appropriate python versions and platform. I'm not sure we've had anyone use that trick.
straight-action-8031807/05/2022, 4:14 PM