https://pantsbuild.org/ logo
#general
Title
# general
a

ancient-france-42909

02/03/2023, 2:56 PM
Is there a way to force a particular manylinux when building a pex? The pex is built on a newer glibc than the one we run it on
e

enough-analyst-54434

02/03/2023, 3:19 PM
You can raise the floor via https://www.pantsbuild.org/docs/reference-python#resolver_manylinux but not lower the ceiling. If you're building for a foreign platforms though, you want to be using
complete_platforms
. Those are exact: https://www.pantsbuild.org/docs/reference-pex_binary#codecomplete_platformscode
a

ancient-france-42909

02/03/2023, 3:22 PM
So, you can request something newer, not something older. I see. Okay, let's see that one
Let me know if those instructions work for you or not.
a

ancient-france-42909

02/03/2023, 3:27 PM
Oh, that's very verbose.
Now I need to get an Amazon Linux 😛
okay, that won't work, we don't have wheels for all packages
e

enough-analyst-54434

02/03/2023, 4:04 PM
Yeah; so the only way to handle that is pre-build wheels or build in a container with a glibc <= to the target environment glibc.
It's glibc that dictates the manylinux version of built wheels. You can't fake it.
a

ancient-france-42909

02/03/2023, 4:12 PM
But, we're not building anything. We have .tar.gz from pypi that are just pure python (but old).
e

enough-analyst-54434

02/03/2023, 4:16 PM
Ah, ok. Well, you need to pre-build that wheel anyhow. Pex does not currently implement "try and build all sdists - who knows, some may turn out not platform specific" for foreign platforms.
Pex purely delegates to Pip here, and when Pip is arranged for a foreign platform it refuses to build wheels.
I don't see any easy way for Pex to try the yolo route above except when using a lock file. In that case this definitely could be done. If that's of interest to you you might file a feature request issue over at https://github.com/pantsbuild/pex/issues/new to get the feature on the radar.
a

ancient-france-42909

02/03/2023, 4:26 PM
But, I don't want pex to try and build anything, I just want it to get a wheel for an older glibc. There would be no guarantee that it'd work (though, we know it does), but yeah. I think pex added support for the platform
_manylinux
module override in a version that's not in 2.14.0, no?
e

enough-analyst-54434

02/03/2023, 4:29 PM
Pex cannot get a wheel for an older glibc. As I said, the manylinux option raises the floor (use no older than) it does not lower the ceiling. Pip will grab the best match possible and there is no knob to change that, except via complete_platforms.
The problem here is your best possible match is a newer glibc than production. The real problem is, of course, not building on the production image. Pants / Pex try to let you get away with that, but there are limits. And that limit for Python is having to have a complete set of pre-built wheels.
In 2.15.x there is: + https://www.pantsbuild.org/v2.15/docs/reference-pex_binary#codeenvironmentcode + https://www.pantsbuild.org/v2.15/docs/reference-docker_environment The new environment feature attempts to allow you to run certain build steps in containers and can help this case. The feature is very new and I know ~nothing about it, but others may be able to help you with it.
That would allow you to build for production ~in production by using a production (or production-compatible) image for builds locally.
Ah, right, here is the intro / high-level doc for the feature: https://www.pantsbuild.org/v2.15/docs/environments
@ancient-france-42909 Pex can now handle the pure Python sdist case for foreign platforms. You'll need to use Pex 2.1.125 by adding this
pants.toml
snippet if you want to try it out:
Copy code
[pex-cli]
version = "v2.1.125"
known_versions = [
    "v2.1.125|macos_arm64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|macos_x86_64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|linux_x86_64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|linux_arm64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732"
]
a

ancient-france-42909

03/03/2023, 10:38 AM
We actually worked around that particular case, but I'm pretty sure this will come up again as we're upgrading from python 3.7 to 3.10, thanks
6 Views