Hi Is it possible to combine the default version of a packag

Question

Hi Is it possible to combine the default version of a package with a local one It has been discussed before but I couldn t find a solution that suits my needs It s about our friend `pytorch` What we would like is to use the regular upstream version e g `1 12 1` for all OSes except Linux where we want to use `1 12 1+cpu` Our issue being that the upstream Linux wheel is very large and we don t use any of the capabilities included in the Linux wheel The CPU one would suit us just fine However as soon as we add an index that contains any local versions it always only select a suitable package for the current platform OS ignoring all the other ones This results in incompatible dependency error on other platforms than the one that generated the lockfile The only solution that seems to work is to use the version upstream pypi but as I said then we unfortunately use the very large wheels for Linux Any recommendations on how to solve circumvent this issue

enough-analyst-54434 · Answer

Are you familiar with environment markers <https peps python org pep 0508 environment markers|https peps python org pep 0508 environment markers> I imagine that s what you need 2 dependencies where only one is selected based on environment

curved-television-6568 · Answer

Hi < modern manchester 33562> That sounds very similar to what I needed just the other day and this was pretty close to a working solution for that just needed to add some version constraints rather than leave them floating

curved-television-6568 · Answer

Oh and you may of course preferably provide the requirements as an `override` on the `python requirements` target rather than single one out like here it was during my testing session only

modern-manchester-33562 · Answer

First of all thank you for your quick respond Yes I ve looked into the environment markers First I tried something like this ```torch==1 11 0 sys platform = linux torch==1 11 0+cpu sys platform== linux ``` But it seems to ignore the first requirement completely and only lock the `1 11 0+cpu` Both Linux and Mac produces the same result So I though maybe I have to always use local versions as they seems to take precedence Since the upstream version is `+cu105` I tried ```torch==1 11 0+cu102 sys platform = linux torch==1 11 0+cpu sys platform== linux ``` but that gives me the following error ```ERROR Cannot install torch==1 11 0+cpu and torch==1 11 0+cu102 because these package versions have conflicting dependencies ERROR ResolutionImpossible for help visit <https pip pypa io en latest user guide fixing conflicting dependencies> The conflict is caused by The user requested torch==1 11 0+cpu The user requested torch==1 11 0+cu102 To fix this you could try to 1 loosen the range of package versions you ve specified 2 remove package versions to allow pip attempt to solve the dependency conflict``` which doesn t make sense to me as they should never be used together I should also say I m using pants version 2 13 0

enough-analyst-54434 · Answer

Ah yeah The 1st issue actually makes sense The lock must pick a version and that presents 2 versions so one wins As for your second attempt and the conflicts there is not much to do about that there are conflicts and you can t fake your way out of that

enough-analyst-54434 · Answer

I think you need to use Pants multiple resolves feature here

enough-analyst-54434 · Answer

In case you have looked into that yet some documentation here <https www pantsbuild org docs python third party dependencies multiple lockfiles>

modern-manchester-33562 · Answer

It is time to revisit this again With the new 1 13 version of pytorch it starts pulling in even more cuda packages We ve seemed to find a manual solution for this where we just bluntly replaces `torch 1 13 1 cp310 cp310 manylinux1 x86 64 whl` in the PEX lockfile with the URL for `torch 1 13 1 2Bcpu cp310 cp310 linux x86 64 whl` This is obviously not your typical solution for dependency resolution but it fits out use case perfectly If Mac install pypi torch 1 13 1 If Linux install <https download pytorch org whl cpu> torch 1 13 1+cpu The question is just how to best integrate this hack into pants thinking face Is there an easy way to intercept the `generate lockfile` goal to override this

enough-analyst-54434 · Answer

So did you try using the multiple resolves avenue mentioned above

enough-analyst-54434 · Answer

I understand you got a hack working and would probably like to continue with that but to explore the whys and the avenues it should be the case that no hacks are needed if you use multiple resolves It may be more inconvenient Unsure but to discuss the avenues

modern-manchester-33562 · Answer

Yes that doesn t work Since the issue occurs when adding the `torch+cpu` index As soon as that version is present in one of the indexes it throws out all other torch versions and only keeps the `+cpu` one AFAIK it is not possible to parametrize the indexes

modern-manchester-33562 · Answer

Local version seems to also have precedence It seems you can never combine regular versions with local versions e g `+cpu`

enough-analyst-54434 · Answer

Yeah the find links and indexes are global Your use case is different from Jacob s but you need the same thing <https github com pantsbuild pants issues 17565>

enough-analyst-54434 · Answer

Ok that s unfortunate This damn Pytorch distribution model is painful

enough-analyst-54434 · Answer

So one option Disable lock file generation by Pants I do this here for different reasons See comment <https github com pantsbuild scie pants blob 83160ea1f69d92c55315d7d21b960cc90565d05f pants toml L41 L46> This would mean you d run `pex3 lock create ` yourself when you needed to update your lock

enough-analyst-54434 · Answer

Pants would just read the lock never generate it That may be too messy not viable though not sure of your case setup

enough-analyst-54434 · Answer

Do you get requirements from a requirements txt Or is it some pyproject toml

modern-manchester-33562 · Answer

Thanks for your input < enough analyst 54434> I ll make sure to observe that issue and also see if it that other option is viable for us

modern-manchester-33562 · Answer

requirements txt

enough-analyst-54434 · Answer

Ah then the pex3 lock create route is pretty straightforward DO you want the command line

modern-manchester-33562 · Answer

Yes please slightly smiling face

modern-manchester-33562 · Answer

How about integrating into the `generate lockfile` goal and doing some custom merging of lockfiles Would that be possible

enough-analyst-54434 · Answer

```pex3 lock create r requirements tx style universal pip version XXX resolver version pip 2020 resolver target system linux target system mac interpreter constraint YYY indent 2 output lock json```

enough-analyst-54434 · Answer

I don t know what XXX and YYY you current have so you fill those in

enough-analyst-54434 · Answer

So the problem with the custom goal is technically the Pex lock file format is opaque Its not supported as read write it s intended read only IOW Pex reserves the right to change the format which would break your goal use

enough-analyst-54434 · Answer

You could certainly do that though and buyer beware

modern-manchester-33562 · Answer

I assumed as much I ll have a look at the command you sent me and try to make something out of that It all boils down to swapping out one artifact E g this ``` algorithm sha256 hash fd12043868a34a8da7d490bf6db66991108b00ffbeecb034228bfcbbd4197143 url ``` to this ``` algorithm sha256 hash 11692523b87c45b79ddfb5148b12a713d85235d399915490d94e079521f7e014 url ``` And keeping the version as `1 13 1` Then everything works perfectly

enough-analyst-54434 · Answer

I don t fully believe you that new hash looks too short for sha256 but gotcha

enough-analyst-54434 · Answer

slightly smiling face

modern-manchester-33562 · Answer

Fixed it slightly smiling face

enough-analyst-54434 · Answer

Immutable artifacts mutable messages

modern-manchester-33562 · Answer

FYI We chose to solve this by setting up our own virtual pypi repository which Contains upstream pypi Contains <https download pytorch org whl cpu> Contains re packaged versions of the macOS PyTorch wheels with `+cpu` suffixed version numbers Now this pins perfectly

enough-analyst-54434 · Answer

That s good to hear but it s also unfortunate A lot of folks will need to do similar It would be good if there were a less labor intensive way to solve this

lemon-oxygen-72498 · Answer

< modern manchester 33562> < enough analyst 54434> gt any update on this solution I m exactly in your situation < modern manchester 33562> and I m not so fund of putting the solution into place sweat smile

enough-analyst-54434 · Answer

< lemon oxygen 72498> I m almost positive no one contributing to Pants development has done any work here yet

lemon-oxygen-72498 · Answer

Thanks for the update John +1

modern-manchester-33562 · Answer

I m more of the opinion that this is something that torch should solve All solutions I ve seen discussed here in the channel just feel so wrong

lemon-oxygen-72498 · Answer

Yeah agreed I feel bad putting the solution you described above into place disappointed