We're building PEXes with one linux image (the github worfklow's

ubuntu-latest

) and running them in another (

python:3.X-slim

). We use these build flags: •

--platform=manylinux2014_x86_64-cp-{version_tag}-cp{version_tag}

where

version_tag

is something like

"38"

•

--resolve-local-platforms

Binary wheels work great which is expected. Since we

--resolve-local-platforms

and we also ensure the target Python interpreter version is present in the builder-image, those have also worked.. so far. I recently saw a problem with one sdist dependency where the PEX was build successfully, but didn't run and failed with:

Needed cp310-cp310-manylinux_2_31_x86_64 compatible dependencies for:
...
1: pendulum

The PEX included the

manylinux_2_35

version instead of the

manylinux_2_31

version for this dependency. I suspect this has to do with the github upgrading their

ubuntu-latest

to from 20.04 to 22.04 (see compatibility). Does my analysis seem correct so far? I'm looking for a solution that lets us reliably build sdists: 1. pin the builder image to ubuntu 20.04. we then rely on github's support for this image 2. run our own docker image during the build - is slow and looking to avoid this option 3. build a backwards compatible sdist. is it even possible to build a

manylinux_2_31

wheel on ubuntu 22.04? 4. any other options?

--platform is a lie, use --complete-platform

I saw another thread re this as well. But do you think that using

complete_platform

we will be able to build a backwards compatible wheel on ubuntu 22.04? I'm not sure how the

manylinux_*

tag is generated.

Right here: https://pex.readthedocs.io/en/v2.1.119/buildingpex.html#complete-platform You generate a complete platform with:

$ docker run --rm -t python:3.10-slim bash -c 'python3.10 -mvenv venv && source venv/bin/activate && pip -q install pex && pex3 interpreter inspect --markers --ta
gs -i 2'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
{
  "path": "/venv/bin/python3.10",
  "compatible_tags": [
    "cp310-cp310-manylinux_2_31_x86_64",
    "cp310-cp310-manylinux_2_30_x86_64",
..
"py31-none-any",
    "py30-none-any"
  ],
  "marker_environment": {
    "implementation_name": "cpython",
    "implementation_version": "3.10.9",
    "os_name": "posix",
    "platform_machine": "x86_64",
    "platform_python_implementation": "CPython",
    "platform_release": "5.15.79.1-microsoft-standard-WSL2",
    "platform_system": "Linux",
    "platform_version": "#1 SMP Wed Nov 23 01:01:46 UTC 2022",
    "python_full_version": "3.10.9",
    "python_version": "3.10",
    "sys_platform": "linux"
  }
}

makes sense. i'll switch to using

complete-platform

. i believe ubuntu 20.04 used glibc 2.31 and it matches what's on our

python:3.x-slim

. however 22.04 uses 2.35 and doesn't match.

there is nothing Pex can do for you if you're not using a compatible glibc to build wheels from sdists with. You must arrange for that.

is there any way to arrange this besides what was provided by the linux distribution?

Just like my pex3 command above, but use pip wheel

Really the only sane thing to do is have a pre-build process that ensure all dependencies are available as wheels for prod Linux.

also agree. our use case though has user code so we can't provide all the wheels since they're not known in advance. looks like building on a known docker image is the way to go.

Yeah, your task is impossible without constraining the user somehow. They must abide by wheel deps or else a limited image range for you to satisfy it seems to me,

thanks - very helpful. we're on 2.1.111 so will upgrade.

Ok, great.

is it possible to determine which sdists need to be built, without having the target linux environment available? i suspect not but wondering if i can run

pex3 lock

on any linux environment and then inspect the output to see if some sdists need to built. if not, we can avoid running the target docker environment.

Yeah, you can run pex3 lock anywhere. If you run with

--no-build

it will exclude sdists. If that leaves nothing (the project is sdist only), the lock should fail. If you don't use

--no-build

the lock will include sdsists, but you can eye-ball the other artifacts and see what you've got. To do this you probably want:

pex3 lock create --style universal --pip-version 22.3 --resolver-version pip-2020-resolver --interpreter-constraint "YouKnoWTheAnswerHere" --target-system linux --indent 2 [requirements]

super useful - thanks, will try it out!

With that you can eyeball the wheels

our target is python-slim which is 2.31. we control this. however we don't control the build environment that well - eg github workflows gives us glibc 2.35 (but used to give us 2.31). worst case we have to run a second docker image under github that provides 2.31 - not great but not terrible.

Ok, great. Yeah, trying to use your image in CI and not GH's is definitely the way to control your destiny. If you can make that faster by pre-building a base image or via other tricks, it will be worth it.

I'm seeing unexpected behavior trying to use

--complete-platform

with

--resolve-local-platforms

. Transcripts below. Generate the complete platform json:

% docker run --platform=linux/amd64 -it python:3.10-slim /bin/bash -c 'pip3 install pex==2.1.119 --quiet --quiet --quiet; pex3 interpreter inspect --markers --tags;' > py310-complete-platform.json

% cat py310-complete-platform.json| jq '.compatible_tags' | grep cp310-manylinux_2_17
  "cp310-cp310-manylinux_2_17_x86_64",

Prepare the builder from an official pypa builder:

% docker run --platform=linux/amd64 -it -v ${PWD}:/curdir <http://quay.io/pypa/manylinux2014_x86_64|quay.io/pypa/manylinux2014_x86_64>            
[root@3a4bc9eae7f5 /]# export PATH=/opt/python/cp310-cp310/bin/:$PATH
[root@3a4bc9eae7f5 /]# python3.10 --version
Python 3.10.9
[root@3a4bc9eae7f5 /]# which pip3
/opt/python/cp310-cp310/bin/pip3
[root@3a4bc9eae7f5 /]# pip3 install pex
Collecting pex
  Downloading pex-2.1.119-py2.py3-none-any.whl (2.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 6.0 MB/s eta 0:00:00
Installing collected packages: pex
Successfully installed pex-2.1.119

[root@3a4bc9eae7f5 curdir]# cd /curdir
[root@3a4bc9eae7f5 curdir]# export _PEX_FILE_LOCK_STYLE=bsd

Expected failure:

--platform

[root@3a4bc9eae7f5 curdir]# pex --python=python3.10 --platform=manylinux2014_x86_64-cp-310-cp310 --pip-version=22.2.2 --resolver-version=pip-2020-resolver pendulum==2.1.2 -o pendulum-a.pex
pid 154 -> /root/.pex/venvs/6c59b447d0c1988f35c5942be176267a84f608a5/34acb08bef9fb9b19fe854dbe3d6bc571944def9/bin/python -sE /root/.pex/venvs/6c59b447d0c1988f35c5942be176267a84f608a5/34acb08bef9fb9b19fe854dbe3d6bc571944def9/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /root/.pex/pip_cache --log /tmp/pex-pip-log.fi5g_vy9/pip.log download --dest /root/.pex/downloads/resolver_download.x1rd_avf/cp310-cp310-manylinux2014_x86_64 --only-binary :all: pendulum==2.1.2 --platform manylinux2014_x86_64 --implementation cp --python-version 310 --abi cp310 --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pendulum==2.1.2 (from versions: 0.3.1, 0.4, 3.0.0a1)
ERROR: No matching distribution found for pendulum==2.1.2

Expected success:

--platform

+

--resolve-local-platforms

[root@3a4bc9eae7f5 curdir]# pex --python=python3.10 --platform=manylinux2014_x86_64-cp-310-cp310 --pip-version=22.2.2 --resolver-version=pip-2020-resolver pendulum==2.1.2 -o pendulum-b.pex --resolve-local-platforms

[root@3a4bc9eae7f5 curdir]# unzip -l pendulum-b.pex | grep 'whl/$'
        0  01-01-1980 00:00   .deps/pendulum-2.1.2-cp310-cp310-manylinux_2_17_x86_64.whl/
        0  01-01-1980 00:00   .deps/python_dateutil-2.8.2-py2.py3-none-any.whl/
        0  01-01-1980 00:00   .deps/pytzdata-2020.1-py2.py3-none-any.whl/
        0  01-01-1980 00:00   .deps/six-1.16.0-py2.py3-none-any.whl/

Unexpected failure:

--complete-platform

+

--resolve-local-platforms

. The platform targeted by the pex above (manylinux_2_17) appears to be in the complete-platform spec.

[root@3a4bc9eae7f5 curdir]# pex --python=python3.10 --complete-platform=py310-complete-platform.json --pip-version=22.2.2 --resolver-version=pip-2020-resolver pendulum==2.1.2 -o pendulum-c.pex --resolve-local-platforms
pid 513 -> /root/.pex/venvs/6c59b447d0c1988f35c5942be176267a84f608a5/34acb08bef9fb9b19fe854dbe3d6bc571944def9/bin/python -sE /root/.pex/venvs/6c59b447d0c1988f35c5942be176267a84f608a5/34acb08bef9fb9b19fe854dbe3d6bc571944def9/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /root/.pex/pip_cache --log /tmp/pex-pip-log.dig57pt8/pip.log download --dest /root/.pex/downloads/resolver_download.pgnh8tee/cp310-cp310-manylinux_2_31_x86_64 --only-binary :all: pendulum==2.1.2 --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pendulum==2.1.2 (from versions: 0.3.1, 0.4, 3.0.0a1)
ERROR: No matching distribution found for pendulum==2.1.2

I then manually removed all

marker_environment

keys in case that was imposing some other constraint. However I still get the same error. What am I missing?

Thanks for fighting through my confusion and fixing Pex here @abundant-autumn-67998