#17626 `package` and `generate-lockfiles` can generate different sha256s for the same python sdist New issue created by danxmoran Describe the bug We modified our CI environment to include Python 3.9 in addition to 3.8, and set things up so that

./pants

itself would run using 3.9 (while our "main" code continued to use 3.8). Afterwards, we started seeing

./pants package

fail with sha256 mismatch errors for one of our requirements:

There was 1 error downloading required artifacts:artifacts
1. clr 0.3.19 from git+<https://github.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160>
    Expected sha256 hash of 88c4fc49003c1594d70aefe9cfbae3e60327a85e52e4608755f52cce2d438eb3 when downloading clr but hashed to 39cdfcc1e1980477476434e6d3d7777bc2bbecff7fcd903f36b4ddb1b17283a5.

@jsirois pointed out that the hash depended on the version of the interpreter building the sdist:

$ curl -sSL <https://github.com/pantsbuild/pex/releases/download/v2.1.114/pex> -O
$ PEX_SCRIPT=pex3 python3.7 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://githu/>
b.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160#egg=clr" --indent 2 -o lock.3.7.json
$ PEX_SCRIPT=pex3 python3.8 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160#egg=clr>" --indent 2 -o lock.3.8.json
$ PEX_SCRIPT=pex3 python3.9 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160#egg=clr>" --indent 2 -o lock.3.9.json
$ diff lock.3.7.json lock.3.8.json
52c52
<               "hash": "39be551f54bc4c59210f08326bc649806d0ca18b8c008975424023cda2568782",
---
>               "hash": "88c4fc49003c1594d70aefe9cfbae3e60327a85e52e4608755f52cce2d438eb3",
$ diff lock.3.7.json lock.3.9.json
52c52
<               "hash": "39be551f54bc4c59210f08326bc649806d0ca18b8c008975424023cda2568782",
---
>               "hash": "39cdfcc1e1980477476434e6d3d7777bc2bbecff7fcd903f36b4ddb1b17283a5",

Despite that, running

./pants generate-lockfiles

under Python 3.9 didn't change the expected hash in our lockfile. Inspecting the

package

and

generate-lockfiles

processes, I found that: 1.

package

was generating a process like

$(which python) ./pex --lock <lockfile> <requirement>...

- in this case the Python 3.9 interpreter was selected, which was not compatible with the interpreter constraints in

<lockfile>

2.

generate-lockfiles

was generating a process that ran in a virtualenv - the virtualenv's interpreter was a symlink pointing at the system's Python 3.8 binary As a user, now that I now the sha256 computation of sdists can depend on the interpreter version, I would expect

./pants package

to locate & use the same interpreter as

generate-lockfiles

(or a "compatible" interpreter for some definition of compatible) to avoid this sort of mismatch. Pants version 2.15.x OS Both Additional info At the pex level, the hash mismatch was caused by the use of

setup-requires

in the requirement's

setup.py

. It sounds like there isn't really a way around the problem within pex itself, so the idea here is to have Pants work around the issue automatically when possible. pantsbuild/pants