https://pantsbuild.org/ logo
Join Slack
Powered by
Hey everyone I have an issue that cropped up a whi...
# general
h
Hey everyone I have an issue that cropped up a while back and I'd just like to try understand a bit more context about what's going on. in my pants.toml i added the following lines to help with building packages for my old raspberry pi:
Copy code
[python-repos]
indexes.add = ["<https://www.piwheels.org/simple>"]
After adding this the builds worked for the 
complete_platforms
as pip was able to find the packages needed. However, a colleague went to generate a new lock file and it was failing! I was getting errors like this for various packages:
Copy code
Expected sha256 hash of 37cddd62d83dc4f72f7c3f3c2bcf2697e89a30efb152079896544a93907733bd when downloading fonttools but hashed to 79b6307086b486c4a0c480cd57555a7386b342b2e4bea48d31e57e0635168d97.
It took me a while to realise that the only thing I had changed was this indexes.add. I'm curious why this error was occurring, and what I need to do to avoid this in the future? Our stopgap for now is to add/remove that index when building for the pi. But ideally we understand this behaviour and implement the correct path forward
h
Noting that the published (and computed) sha256 for https://files.pythonhosted.org/packages/57/5e/de2e6e51cb6894f2f2bc2641f6c845561361b622e96df3cca04df77222c9/fonttools-4.54.1-py3-none-any.whl is 37cddd62d83dc4f72f7c3f3c2bcf2697e89a30efb152079896544a93907733bd but the computed sha256 of the file of the same name on the other index (https://www.piwheels.org/simple/fonttools/fonttools-4.54.1-py3-none-any.whl) is indeed 79b6307086b486c4a0c480cd57555a7386b342b2e4bea48d31e57e0635168d97
the version on piwheels has slightly different METADATA than the canonical version on pypi
which version of pants is this?
I assume pex is reading the sha256 file from pypi (because piwheels doesn’t publish those), or relying on previously computed sha256,
so naturally there is a conflict
There is something pretty broken - dangerous even - about an index serving a different version of ostensibly the same artifact, but I guess this is the reality.
(in practice in this case at least the only different is removal of spaces before semicolons in the METADATA, e.g.,
Copy code
< Requires-Dist: fs <3,>=2.2.0 ; extra == 'ufo'
---
> Requires-Dist: fs<3,>=2.2.0; extra == "ufo"
)
But anyway, knowing which pants version this is will be helpful
h
I'm using 2.19.1. Could you provide some context around why its comparing versions between pypi and piwheels and why that's even needed? surely if a version matches on pypi there's no need for it to use pywheels at all? I'm interested to know what's going on behind the scenes here a bit and why those shas are needed etc.
(sorry and of course thank you for the reply and looking into this, i'm very grateful for the amazing support you guys provide)
h
Oh, there’s probably a bug of some kind, I’m just providing context that it may be the case that an index providing different versions of ostensibly the same artifact is bad. But so much of the python packaging ecosystem is bad in these sorts of ways that we must handle it gracefully…
What are the steps to reproduce this? I.e., which platform were you and your coworker creating the lockfile on, and what was your complete_platforms?
If you could create a small repo that reproduces the problem (see bullet #2 here ) that would be great
“Packages are expected to be unique up to name and version, so two wheels with the same package name and version are treated as indistinguishable by pip. This is a deliberate feature of the package metadata, and not likely to change.”
“treated as indistinguishable”… but these are distinguishable (by their differing sha256es)
Python packaging is a mess
However in this case, since presumably this all works with raw pip, it should be made to work with pants/pex too. So step one is a reproduction.
h
So I've tried to reproduce this in a small repo and I've copied the identical pants.toml and requirements-default.txt and run the generate-lock command and it doesn't actually create a full lock. Just what looks like to be a placeholder. How do I force it to actually lock all the requirements?
So once I figure that out, hopefully i can generate the whole lock file and send through the example.
h
There is no “placeholder” as a concept, lockfiles are generated off all their inputs, so presumably the lock inputs in this smaller repo aren’t what you think they are. Maybe a missing BUILD file stanza or something.
h
so the input isn't just resolving the dependencies from the requirements.txt?
h
It should be, but there needs to be a 
python_requirements
target pointing to it in some BUILD file
h
Alright that's what the issue was thank you. I'm generating the lock now and I"ll see if the issue crops up
Okay so I haven't changed a thing and for some reason its now generating the lock file both in the main repo and demo repo 🤷
Open in Slack
PreviousNext
Powered by