modern-manchester-33562
09/15/2022, 2:54 PMpytorch
. What we would like is to use the regular upstream version e.g. 1.12.1
for all OSes except Linux, where we want to use 1.12.1+cpu
. Our issue being that the upstream Linux wheel is very large and we don't use any of the capabilities included in the Linux wheel. The CPU one would suit us just fine. However, as soon as we add an index that contains any local versions it always only select a suitable package for the current platform (OS), ignoring all the other ones. This results in incompatible dependency error on other platforms than the one that generated the lockfile. The only solution that seems to work is to use the version upstream (pypi) but, as I said, then we unfortunately use the very large wheels for Linux. Any recommendations on how to solve / circumvent this issue?enough-analyst-54434
09/15/2022, 3:11 PMcurved-television-6568
09/15/2022, 4:02 PMoverride
on the python_requirements
target rather than single one out like here, it was during my testing session only..modern-manchester-33562
09/16/2022, 6:56 AMtorch==1.11.0; sys_platform!="linux"
torch==1.11.0+cpu; sys_platform=="linux"
But it seems to ignore the first requirement completely and only lock the 1.11.0+cpu
. Both Linux and Mac produces the same result. So I though, maybe I have to always use local versions as they seems to take precedence. Since the upstream version is +cu105
I tried:
torch==1.11.0+cu102; sys_platform!="linux"
torch==1.11.0+cpu; sys_platform=="linux"
but that gives me the following error:
ERROR: Cannot install torch==1.11.0+cpu and torch==1.11.0+cu102 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
The conflict is caused by:
The user requested torch==1.11.0+cpu
The user requested torch==1.11.0+cu102
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
which doesn't make sense to me as they should never be used together.
I should also say I'm using pants version 2.13.0.enough-analyst-54434
09/16/2022, 2:53 PMmodern-manchester-33562
12/16/2022, 2:58 PMtorch-1.13.1-cp310-cp310-manylinux1_x86_64.whl
in the PEX lockfile with the URL for torch-1.13.1%2Bcpu-cp310-cp310-linux_x86_64.whl
. This is obviously not your typical solution for dependency resolution but it fits out use case perfectly:
⢠If Mac, install pypi torch 1.13.1
⢠If Linux, install https://download.pytorch.org/whl/cpu torch 1.13.1+cpu
The question is just how to best integrate this "hack" into pants š¤ Is there an easy way to intercept the generate-lockfile
goal to override this?enough-analyst-54434
12/16/2022, 3:52 PMmodern-manchester-33562
12/16/2022, 4:04 PMtorch+cpu
index. As soon as that version is present in one of the indexes, it throws out all other torch versions and only keeps the +cpu
one.
AFAIK it is not possible to parametrize the indexes?+cpu
enough-analyst-54434
12/16/2022, 4:11 PMpex3 lock create ...
yourself when you needed to update your lock.modern-manchester-33562
12/16/2022, 4:16 PMenough-analyst-54434
12/16/2022, 4:17 PMmodern-manchester-33562
12/16/2022, 4:17 PMgenerate-lockfile
goal and doing some custom merging of lockfiles? Would that be possible?enough-analyst-54434
12/16/2022, 4:18 PMpex3 lock create -r requirements.tx --style universal --pip-version XXX --resolver-version pip-2020-resolver --target-system linux --target-system mac --interpreter-constraint YYY --indent 2 --output lock.json
modern-manchester-33562
12/16/2022, 4:30 PM{
"algorithm": "sha256",
"hash": "fd12043868a34a8da7d490bf6db66991108b00ffbeecb034228bfcbbd4197143",
"url": "<https://files.pythonhosted.org/packages/81/58/431fd405855553af1a98091848cf97741302416b01462bbf9909d3c422b3/torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl>"
}
to this
{
"algorithm": "sha256",
"hash": "11692523b87c45b79ddfb5148b12a713d85235d399915490d94e079521f7e014",
"url": "<https://download.pytorch.org/whl/cpu/torch-1.13.1%2Bcpu-cp310-cp310-linux_x86_64.whl>"
}
And keeping the version as 1.13.1
. Then everything works "perfectly".enough-analyst-54434
12/16/2022, 4:32 PMmodern-manchester-33562
12/16/2022, 4:33 PMenough-analyst-54434
12/16/2022, 4:33 PMmodern-manchester-33562
01/02/2023, 3:32 PM+cpu
suffixed version numbers.
Now this pins perfectly!enough-analyst-54434
01/02/2023, 4:10 PMlemon-oxygen-72498
02/10/2023, 9:15 AMenough-analyst-54434
02/10/2023, 2:25 PMlemon-oxygen-72498
02/10/2023, 3:24 PMmodern-manchester-33562
02/13/2023, 10:22 AMlemon-oxygen-72498
02/17/2023, 7:55 AM