https://pantsbuild.org/ logo
Join Slack
Powered by
trying to debug a puzzler around sha256 mismatches...
# development
s
trying to debug a puzzler around sha256 mismatches when packaging a pex using python 3.9 🧵
we’re trying to switch pants itself to running with python 3.9, while continuing to lint/test/package the main code for python 3.8 (for now). to support that, I extended our CI image to include both versions of python, and to have 
python
point at the 3.9 interpreter. after making that switch, we consistently see one 
./pants package
of a PEX in our build fail with:
Copy code
There was 1 error downloading required artifacts:artifacts
1. clr 0.3.19 from git+<https://github.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160>
    Expected sha256 hash of 88c4fc49003c1594d70aefe9cfbae3e60327a85e52e4608755f52cce2d438eb3 when downloading clr but hashed to 39cdfcc1e1980477476434e6d3d7777bc2bbecff7fcd903f36b4ddb1b17283a5.
I’m not able to repro the issue natively on my mac, but I can repro it consistently if I run the CI container locally
if I keep sandboxes and edit the 
__run.sh
after the failure to point at 3.8, everything starts working
does this sort of error on 3.9 sound familiar to anyone? setting 
PEX_VERBOSE=1
doesn’t give me much more information
e
It looks like the hash depends on the interpreter building the sdist:
Copy code
$ curl -sSL <https://github.com/pantsbuild/pex/releases/download/v2.1.114/pex> -O
$ PEX_SCRIPT=pex3 python3.7 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://githu>
b.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160#egg=clr" --indent 2 -o lock.3.7.json
$ PEX_SCRIPT=pex3 python3.8 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160#egg=clr>" --indent 2 -o lock.3.8.json
$ PEX_SCRIPT=pex3 python3.9 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160#egg=clr>" --indent 2 -o lock.3.9.json
$ diff lock.3.7.json lock.3.8.json
52c52
<               "hash": "39be551f54bc4c59210f08326bc649806d0ca18b8c008975424023cda2568782",
---
>               "hash": "88c4fc49003c1594d70aefe9cfbae3e60327a85e52e4608755f52cce2d438eb3",
$ diff lock.3.7.json lock.3.9.json
52c52
<               "hash": "39be551f54bc4c59210f08326bc649806d0ca18b8c008975424023cda2568782",
---
>               "hash": "39cdfcc1e1980477476434e6d3d7777bc2bbecff7fcd903f36b4ddb1b17283a5",
It's only consistent within 1 interpreter:
Copy code
$ rm -rf ~/.pex
$ PEX_SCRIPT=pex3 python3.7 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/color/clr.git@a444a0612374a19f22e511b5e3514a60561e6160#egg=clr>" --indent 2 -o lock.3.7-2.json
$ diff lock.3.7.json lock.3.7-2.json
🤯 1
Pex goes through some lengths here to get a consistent sdist hash, namely building the sdist and then unpacking it to a diretory, and then content hashing that directory so that tar-ness doesn't throw things off. Clearly there is some other environment sensitive bit in there.
s
putting that surprise aside, it’s also then confusing that 
generate-lockfiles
doesn’t change the expected hash in the lockfile when pants is running under python 3.9. looking at the 
ps
output of a running 
generate-lockfiles
, it appears to be running out of a 3.8 virtualenv. is there some extra “find a compatible interpreter step” in the lockfile-generation logic that we should copy over to the pex-packaging rule so the two are consistent?
e
Well, there is a long way between the high level "Pants is running under 3.9" And Pants rules are also using Python 3.9 to run Pex.
So, when I use 
pip download
to get those sdists the resulting zips differ, when unpacked and the .git dir is ignored, which Pex does:
Copy code
$ diff -x .git -r 3.7.dists/clr 3.9.dists/clr
Only in 3.7.dists/clr/.eggs: pytest_runner-6.0.0-py3.7.egg
Only in 3.9.dists/clr/.eggs: pytest_runner-6.0.0-py3.9.egg
So that appears to be the underlying issue. Looking deeper to see why integration tests don't hit that particular issue, but the color/clr does.
đź‘Ť 1
Ah, its the setup-requires
So ... geez
Not sure I can do anything about that!
Can you not use PEP-517/518 ?
Just drop in a thin pyproject.toml and move the setup-requires there?
s
to unblock us on this issue we should be able to - this is one of our effectively-abandoned projects that we haven’t shaken off yet
e
I'm not seeing any viable way to deal with setup-requires.
I'll test to make sure PEPifying works here.
s
./pants package
ultimately ends up generating/running a process that runs:
Copy code
<python> ./pex <bunch-of-requirements> --lock <lockfile>
if we enforced that the 
<python>
injected was compatible with the interpreter constraints of 
<lockfile>
, would that “fix” it for the 
./pants package
use-case? avoid the underlying issue in pex
e
Perhaps. This all gets super-complex and I'd need to page in alot to be sure of things. I haven't looked at any of that stuff in a while.
đź‘Ť 1
s
I’ll file an issue so we have a more permanent marker for the idea. thanks for the debugging help!
e
Copy code
$ curl -sSL <https://github.com/pantsbuild/pex/releases/download/v2.1.114/pex> -O
$ PEX_SCRIPT=pex3 python3.7 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/jsirois/clr.git@07d525764e6589a1059ee3977bbaefbb687c629a#egg=clr>" --indent 2 -o lock.3.7.json
$ PEX_SCRIPT=pex3 python3.8 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/jsirois/clr.git@07d525764e6589a1059ee3977bbaefbb687c629a#egg=clr>" --indent 2 -o lock.3.8.json
$ PEX_SCRIPT=pex3 python3.9 pex lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver "git+<https://github.com/jsirois/clr.git@07d525764e6589a1059ee3977bbaefbb687c629a#egg=clr>" --indent 2 -o lock.3.9.json
$ diff lock.3.7.json lock.3.8.json
$ diff lock.3.7.json lock.3.9.json
$
Python packaging is an unholy mess.
That PR breaks pytest-runner though. But also, read the deprecation notice 🤦 https://pypi.org/project/pytest-runner/
Basically, the color/clr recommended means of running tests is broken. You need to either not run tests or invest a smidgen in modernizing the test running, maybe using tox like the pytest-runner deprecation notice suggests.
s
we have been wanting to fold this package into the monorepo and build it with pants for awhile - maybe this will be the straw that finally breaks the camel’s back
nobody’s touched the repo for ~a year so deleting/disabling tests may also be a viable option in the short-term
e
As to the Pants bug, I'm just not sure there is anything that can be done here. I commented on the issue.
Open in Slack
PreviousNext
Powered by