https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to force a particular manylinux whe...
# general
a
Is there a way to force a particular manylinux when building a pex? The pex is built on a newer glibc than the one we run it on
e
You can raise the floor via https://www.pantsbuild.org/docs/reference-python#resolver_manylinux but not lower the ceiling. If you're building for a foreign platforms though, you want to be using 
complete_platforms
. Those are exact: https://www.pantsbuild.org/docs/reference-pex_binary#codecomplete_platformscode
a
So, you can request something newer, not something older. I see. Okay, let's see that one
e
Let me know if those instructions work for you or not.
a
Oh, that's very verbose.
Now I need to get an Amazon Linux 😛
okay, that won't work, we don't have wheels for all packages
e
Yeah; so the only way to handle that is pre-build wheels or build in a container with a glibc <= to the target environment glibc.
It's glibc that dictates the manylinux version of built wheels. You can't fake it.
a
But, we're not building anything. We have .tar.gz from pypi that are just pure python (but old).
e
Ah, ok. Well, you need to pre-build that wheel anyhow. Pex does not currently implement "try and build all sdists - who knows, some may turn out not platform specific" for foreign platforms.
Pex purely delegates to Pip here, and when Pip is arranged for a foreign platform it refuses to build wheels.
I don't see any easy way for Pex to try the yolo route above except when using a lock file. In that case this definitely could be done. If that's of interest to you you might file a feature request issue over at https://github.com/pantsbuild/pex/issues/new to get the feature on the radar.
a
But, I don't want pex to try and build anything, I just want it to get a wheel for an older glibc. There would be no guarantee that it'd work (though, we know it does), but yeah. I think pex added support for the platform 
_manylinux
module override in a version that's not in 2.14.0, no?
e
Pex cannot get a wheel for an older glibc. As I said, the manylinux option raises the floor (use no older than) it does not lower the ceiling. Pip will grab the best match possible and there is no knob to change that, except via complete_platforms.
The problem here is your best possible match is a newer glibc than production. The real problem is, of course, not building on the production image. Pants / Pex try to let you get away with that, but there are limits. And that limit for Python is having to have a complete set of pre-built wheels.
In 2.15.x there is: + https://www.pantsbuild.org/v2.15/docs/reference-pex_binary#codeenvironmentcode + https://www.pantsbuild.org/v2.15/docs/reference-docker_environment The new environment feature attempts to allow you to run certain build steps in containers and can help this case. The feature is very new and I know ~nothing about it, but others may be able to help you with it.
That would allow you to build for production ~in production by using a production (or production-compatible) image for builds locally.
Ah, right, here is the intro / high-level doc for the feature: https://www.pantsbuild.org/v2.15/docs/environments
@ancient-france-42909 Pex can now handle the pure Python sdist case for foreign platforms. You'll need to use Pex 2.1.125 by adding this 
pants.toml
snippet if you want to try it out:
Copy code
[pex-cli]
version = "v2.1.125"
known_versions = [
    "v2.1.125|macos_arm64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|macos_x86_64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|linux_x86_64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|linux_arm64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732"
]
a
We actually worked around that particular case, but I'm pretty sure this will come up again as we're upgrading from python 3.7 to 3.10, thanks
6 Views
Open in Slack
PreviousNext
Powered by