Running into an interesting lockfile generation issue with `

Question

Running into an interesting lockfile generation issue with `torch` package thread

rhythmic-battery-45198 · Answer

Was seeing the following error raised after adding a `torch` dependency ```stderr Failed to resolve requirements from PEX environment home ci cache pants named caches pex root unzipped pexes a5788e2aff6bc00ba7d5959665bfbcc6e8421d1c Needed cp39 cp39 manylinux 2 35 x86 64 compatible dependencies for 1 nvidia cuda runtime cu11==11 7 99 platform system == Linux Required by torch 1 13 1 But this pex had no ProjectName raw= nvidia cuda runtime cu11 normalized= nvidia cuda runtime cu11 distributions 2 nvidia cudnn cu11==8 5 0 96 platform system == Linux Required by torch 1 13 1 But this pex had no ProjectName raw= nvidia cudnn cu11 normalized= nvidia cudnn cu11 distributions 3 nvidia cublas cu11==11 10 3 66 platform system == Linux Required by torch 1 13 1 But this pex had no ProjectName raw= nvidia cublas cu11 normalized= nvidia cublas cu11 distributions 4 nvidia cuda nvrtc cu11==11 7 99 platform system == Linux Required by torch 1 13 1 But this pex had no ProjectName raw= nvidia cuda nvrtc cu11 normalized= nvidia cuda nvrtc cu11 distributions ```

rhythmic-battery-45198 · Answer

Checked the lockfile As expected those nvidia distributions were missing ``` requires dists opt einsum gt =3 3 extra == opt einsum typing extensions ```

rhythmic-battery-45198 · Answer

Did some pex poking around Isolated the interesting behavior

rhythmic-battery-45198 · Answer

With ` style universal` and ` target system linux` options set the lockfile is generated without the expected nvidia libs This is the configuration used by pants when the error is raised ``` pex3 lock create style=universal resolver version pip 2020 resolver target system linux torch==1 13 1```

rhythmic-battery-45198 · Answer

If I instead use the default ` style strict` on my linux x86 64 machine the nvidia distributions make it into the lockfile ```pex3 lock create style=strict resolver version pip 2020 resolver torch==1 13 1``` ``` requires dists nvidia cublas cu11==11 10 3 66 platform system == Linux nvidia cuda nvrtc cu11==11 7 99 platform system == Linux nvidia cuda runtime cu11==11 7 99 platform system == Linux nvidia cudnn cu11==8 5 0 96 platform system == Linux opt einsum gt =3 3 extra == opt einsum typing extensions ```

rhythmic-battery-45198 · Answer

The root issue seems to be that different `torch` wheels list different transitive dependencies It looks like the transitive dependencies are picked up from the first artifact listed in the lockfile which looks like the same order that is listed on pypi <https pypi org project torch 1 13 1 files>

rhythmic-battery-45198 · Answer

With universal style <https files pythonhosted org packages 86 08 41315a205bcd103a9698fa8afafbb73a234db8791cdb8b96d1efb10243a7 torch 1 13 1 cp39 cp39 manylinux2014 aarch64 whl|https files pythonhosted org packages 86 08 41315a205bcd103a9698fa8afafbb73a234db8791c b10243a7 torch 1 13 1 cp39 cp39 manylinux2014 aarch64 whl> is the first artifact This whl lacks nvidia dependencies in its `METADATA` file ```Classifier Programming Language C++ Classifier Programming Language Python 3 Classifier Programming Language Python 3 7 Classifier Programming Language Python 3 8 Classifier Programming Language Python 3 9 Classifier Programming Language Python 3 10 Requires Python gt =3 7 0 Description Content Type text markdown License File LICENSE License File NOTICE Requires Dist typing extensions Provides Extra opt einsum Requires Dist opt einsum gt =3 3 extra == opt einsum ```

rhythmic-battery-45198 · Answer

With strict style <https files pythonhosted org packages 81 58 431fd405855553af1a98091848cf97741302416b01462bbf9909d3c422b3 torch 1 13 1 cp310 cp310 manylinux1 x86 64 whl|https files pythonhosted org packages 81 58 431fd405855553af1a98091848cf97741302416b01 09d3c422b3 torch 1 13 1 cp310 cp310 manylinux1 x86 64 whl> is the artifact This whl does include nvidia dependencies in its `METADATA` file ```Classifier Programming Language C++ Classifier Programming Language Python 3 Classifier Programming Language Python 3 7 Classifier Programming Language Python 3 8 Classifier Programming Language Python 3 9 Classifier Programming Language Python 3 10 Requires Python gt =3 7 0 Description Content Type text markdown License File LICENSE License File NOTICE Requires Dist typing extensions Requires Dist nvidia cuda runtime cu11 ==11 7 99 platform system == Linux Requires Dist nvidia cudnn cu11 ==8 5 0 96 platform system == Linux Requires Dist nvidia cublas cu11 ==11 10 3 66 platform system == Linux Requires Dist nvidia cuda nvrtc cu11 ==11 7 99 platform system == Linux Provides Extra opt einsum Requires Dist opt einsum gt =3 3 extra == opt einsum ```

rhythmic-battery-45198 · Answer

I can probably work around this easily by adding explicit dependencies for the `torch` `python requirement` <https www pantsbuild org docs python third party dependencies requirements with undeclared dependencies>

rhythmic-battery-45198 · Answer

I am wondering if this is a just a misbehaving package not sure if all wheels should list the same dependencies something that I can configure pants pex to handle today a case not currently handled by pants pex end

enough-analyst-54434 · Answer

A core assumption to make locks feasible at all is the assumption you ve discovered all distributions for a given version sdist or whl have the same deps Without that assumption you d be forced to download every wheel for a given version and that is prohibitive in time and bandwidth The discourse thread for this rejected PEP talks about this necessary assumption <https peps python org pep 0665 >

enough-analyst-54434 · Answer

Really the assumption has nothing to do with making locks feasible it has to do with making resolving in general feasible

enough-analyst-54434 · Answer

So yeah this is really torch doing something legal that is however hostile to tooling

enough-analyst-54434 · Answer

< rhythmic battery 45198> hopefully you can work around as you suggested I really have no ideas on how to approach a fix for this It would require making Pip work in a way that it does not currently

enough-analyst-54434 · Answer

They use environment markers so the state of those wheels is a bit strange They could have uniform dependency metadata with more use of environment markers So its not that the maintainers reject the use of environment markers It seems more like they are unaware the pain they are causing or not sure

rhythmic-battery-45198 · Answer

gt Ok thanks for the background That all makes sense to me and is what I was suspecting Workaround should be pretty painless

enough-analyst-54434 · Answer

Thanks for digging on that one Definitely an interesting case Concerning too Torch is pretty popular This is going to be a wider problem

gentle-painting-24549 · Answer

I ran into the same issue here and resolved it by manually overriding my lock file s `torch requires dists` array to this ``` project name torch requires dists opt einsum gt =3 3 extra == opt einsum typing extensions nvidia cublas cu11==11 10 3 66 platform system == Linux nvidia cuda nvrtc cu11==11 7 99 platform system == Linux nvidia cuda runtime cu11==11 7 99 platform system == Linux nvidia cudnn cu11==8 5 0 96 platform system == Linux ``` I m testing out using dependency overrides in my BUILD files instead now to see if I can do that instead Curious if you ever got this to work < rhythmic battery 45198>

gentle-painting-24549 · Answer

Side note I m seeing those dependencies exposed from PyPI s JSON endpoint I wonder what PyTorch is doing to expose them there to not also expose that information to where pants needs it ```curl s <https pypi org pypi torch 1 13 1 json> | jq r info requires dist ```

enough-analyst-54434 · Answer

The json endpoint has the same problem I d guess If the METADATA is not the same in each wheel it picks one and displays it And they picked one you happen to like

enough-analyst-54434 · Answer

That might change tomorrow if some you d think unimportant sorting changes New wheel gets picked as the random provider of METADATA for that version and you lose

enough-analyst-54434 · Answer

I remember a thread with Donald Stufft where he points out this problem He maintains PyPI

rhythmic-battery-45198 · Answer

I added the nvidia packages to my requirements file and these overrides to my python requirements target ``` overrides= torch dependencies nvidia cuda runtime cu11 nvidia cudnn cu11 nvidia cublas cu11 nvidia cuda nvrtc cu11 ```

gentle-painting-24549 · Answer

Perfect this worked for me and is much better than manually editing my Lock file Thank you It s not actually an issue but I m surprised that pants doesn t inject these dependencies into the underlying `setup py` files for the resulting python distribution `torch==1 13 x` is the only dependency that shows up there

gentle-painting-24549 · Answer

Even more painful this list of requirements needs to be carefully maintained when upgrading to new torch versions Looks like they are doubling the size of nvidia packages needed on the next release <https github com pytorch pytorch pull 89944> It looks like they inject dependencies at the wheel level instead of the package level since since they build CUDA ROCm and CPU versions of the wheels

adamant-magazine-16751 · Answer

Hi I faced this issue today I see that it has already been discussed The option to manually specify all requirements seems to solve the problem but I m looking for a potential way to simplify it I m don t mind building and running only on linux I saw that there are parameters to `pex lock create` that control the lockfile generation namely ` style=strict` and ` target system=linux` I couldn t find a way to set this in pants though I m also not sure what they do exactly and what are the potential caveats of using them Is it documented somewhere So for example would running `pex lock create style=strict` on CentOS make it unusable on other distros due to some manylinux compatibility issues I m sorry if this sound silly but I m not well read in this area sweat

enough-analyst-54434 · Answer

Yeah Pants does not allow you to pick lock style or the list of target systems If it did the strict style would have the problem you guessed As to documentation there is just CLI help for locking ```$ pex venv bin pex3 lock create help usage pex3 lock create h style strict sources universal target system linux mac windows path mapping PATH MAPPINGS o PATH indent INDENT r FILE or URL constraints FILE or URL python PYTHON python path PYTHON PATH interpreter constraint INTERPRETER CONSTRAINT platform PLATFORMS complete platform COMPLETE PLATFORMS manylinux ASSUME MANYLINUX resolve local platforms resolver version pip legacy resolver pip 2020 resolver pip version vendored 20 3 4 patched 22 2 2 22 3 22 3 1 23 0 23 0 1 allow pip version fallback pypi f PATH URL i URL retries RETRIES timeout SECS proxy PROXY cert PATH client cert PATH cache ttl DEPRECATED H DEPRECATED pre wheel build prefer wheel force pep517 build isolation transitive j JOBS preserve pip download log v emit warnings pex root PEX ROOT disable cache cache dir CACHE DIR tmpdir TMPDIR rcfile RC FILE requirements optional arguments h help show this help message and exit style strict sources universal The style of lock to generate The strict style is the default and generates a lock file that contains exactly the distributions that would be used in a local PEX build If an sdist would be used the sdist is included but if a wheel would be used an accompanying sdist will not be included The sources style includes locks containing both wheels and the associated sdists when available The universal style generates a universal lock for all possible target interpreters and platforms although the scope can be constrained via one or more interpreter constraint Of the three lock styles only strict can give you full confidence in the lock since it includes exactly the artifacts that are included in the local PEX you ll build to test the lock result with before checking in the lock With the other two styles you lock un vetted artifacts in addition to the strict ones so even though you can be sure to reproducibly resolve those same un vetted artifacts in the future they re still un vetted and could be innocently or maliciously different from the strict artifacts you can locally vet before committing the lock to version control The effects of the differences could range from failing a resolve using the lock when the un vetted artifacts have different dependencies from their sibling artifacts to your application crashing due to different code in the sibling artifacts to being compromised by differing code in the sibling artifacts So although the more permissive lock styles will allow the lock to work on a wider range of machines are apparently more convenient the convenience comes with a potential price and using these styles should be considered carefully target system linux mac windows The target operating systems to generate the lock for This option applies only to ` style universal` locks and restricts the locked artifacts to those compatible with the specified target operating systems By default universal style locks include artifacts for all operating systems ```

gentle-painting-24549 · Answer

upside down face back again with this issue for a completely different package `open3d` The dependencies you ll find at `curl s <https pypi org pypi open3d 0 17 0 json> | jq r info requires dist ` are wildly different than the dependencies on the wheel we need `open3d 0 17 0 cp38 cp38 manylinux 2 27 x86 64 whl` In our case the results from `generate lockfiles` changed in a matter of days I believe this was triggered by a new wheel upload to PyPI leading to a breaking change where it removed a large number of packages from a lock file

enough-analyst-54434 · Answer

Not much I know how to do What are you hoping for < gentle painting 24549>

gentle-painting-24549 · Answer

Oh nothing at all I was able to resolve the issue using `overrides` Just wanted to leave a note about it here in case someone runs into the same issue with open3d The issue was super perplexing for the team it was very useful to have known about this

enough-analyst-54434 · Answer

Gotcha