https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to generate a platform-specific pex...
# general
b
Is there a way to generate a platform-specific pex, while not running the package command on on the target platform? Details in 🧵
In our specific case, we’re getting an error message like this when running on MacOS CI build nodes that have Cataline (10.16) installed:
Copy code
10:00:50  Needed macosx_10_16_x86_64-cp-39-cp39 compatible dependencies for:
10:00:50   1: wrapt<2,>=1.10
10:00:50      Required by:
10:00:50        Deprecated 1.2.11
10:00:50      But this pex had no 'wrapt' distributions.
If I package the pex on a Catalina machine, then everything works. If I package it on Big Sur, it fails to execute the pex with the error above.
And, if I attempt to package on Big Sur (or inside an docker container running ubuntu) with 
platforms=["macosx_10_15_x86_64-cp39-cp39"]
added to the build target, the package command fails with:
Copy code
ERROR: Could not find a version that satisfies the requirement wrapt<2,>=1.10 (from deprecated)
ERROR: No matching distribution found for wrapt<2,>=1.10
This wrapt transitive dep has been a thorn in our side for a while now, so we might even try to get rid of the dependency that’s pulling it in, but I’m also wondering whether we can actually resolve this packaging issue.
h
Hello!
Is there a way to generate a platform-specific pex, while not running the package command on on the target platform?
You can, but there must be prebuilt wheels for the platform you're targeting. It looks like it does not have that: https://pypi.org/project/wrapt/#files So, you can only get a wheel from whatever the current platform is. If you're building on macOS Catalina, the wheel will have that platform. If Big Sur, it will have that. I am surprised that a Pex with Big Sur wheels is not working on Catalina, though. My understanding of macOS platforms is that they're forward compatible
b
Thanks Eric! Where does it get the wheel for the current platform? Is it somehow built at package time?
h
Yes, precisely. Because there is only an sdist (source distribution) available on PyPI, Pex/pip will build that sdist into a bdist / .whl file
b
Got it. Do you have experience with reaching out to project owners and asking them to upload a pre-built .whl on pypi? Is that a reasonable request?
h
It looks like they have a C native extension, so it isn't super trivial for them to do, but also lots of prior art. They would need to build a wheel for each Python they support x OS, so macOS-cp27 for example. (They can use a stable ABI so that you only need one wheel for Python 3 to work for each OS) They'd probably be more receptive if you offered to do it. As with any open source project, framing it like "what would you think about this? Would you be open to me helping with this?" rather than "my build is broken. Please fix" will take you a long way https://pythonwheels.com is a good website to talk about why it's great for libraries to release wheels instead of only sdists
👍 1
Another approach is for your org to build the wheel and host it in an "internal artifactory". This is how Pants does our release - we build our wheels in Centos7 for maximum compatibility, and then fetch those prebuilt wheels for the release
🧠 1
Open in Slack
PreviousNext
Powered by