#17621 python_awslambda can include distributions incompatible with explicit complete_platforms New issue created by huonw Describe the bug We have a Python lambda built via

python_awslambda

(very convenient, thanks!) plus a complete platform JSON that we got by running

pex3 interpreter inspect --markers --tags

from within the Lambda environment. This environment uses tags up to

"cp39-cp39-manylinux_2_26_x86_64"

. When building on certain Linux systems, it seems that pants will happily include a

manylinux_2_28

wheel (NB. 28 vs 26), which isn't compatible with the lambda environment. Pex thus fails at start-up:

...
pex: Unresolved requirements:
pex:   - cryptography>=3.4.0; extra == "cryptography"
pex: Distributions contained within this pex:
...
pex:   - cryptography-38.0.3-cp36-abi3-manylinux_2_28_x86_64.whl
...
Failed to import module 'lambdex_handler': Failed to resolve requirements from PEX environment @ /var/task.
Needed cp39-cp39-manylinux_2_26_x86_64 compatible dependencies for:
 1: cryptography>=3.4.0; extra == "cryptography"
    Required by:
      python-jose 3.3.0
    But this pex had no ProjectName(raw='cryptography', normalized='cryptography') distributions.
...

I've tried to create a reproducer in https://gist.github.com/huonw/504ef2a5d44efdc2c2df290c0ce5df16. This uses a complete platform from the Python 3.9 lambda environment, with a dependency that doesn't have any matching wheels. (Unfortunately due to https://pantsbuild.slack.com/archives/C046T6T9U/p1669150590877779, I cannot offer scripted commands that definitely reproduce it, because I cannot run pants under docker on my machine at the moment.) The following shell snippet loads the repo, and packages the lambda and checks what is included:

git clone <https://gist.github.com/504ef2a5d44efdc2c2df290c0ce5df16.git> && cd 504ef2a5d44efdc2c2df290c0ce5df16 && ./pants package :lambda && unzip -p dist/lambda.zip PEX-INFO | jq .

The

distributions

object should include a

cryptography

distribution that's compatible with the complete platforms, e.g.

cryptography-38.0.3-cp36-abi3-manylinux_2_24_x86_64.whl

. 1. ✅ it does on macOS 2. ✅ it does on Linux with Python version != 3.9 3. ❌ it doesn't on (some) Linux on Python version 3.9 (it uses

cryptography-38.0.3-cp36-abi3-manylinux_2_28_x86_64.whl

). The specific environment is Github Actions

ubuntu-latest

with

actions/setup-python@v4

used with

python-version: 3.9

. Complete platform in "Additional info" below. I suspect the general behaviour here is that if the host system is close enough to the lambda environment, then pants/src/python/pants/backend/awslambda/python/rules.py Lines 107 to 109 in </pantsbuild/pants/commit/a21d8cb9b90289b1ef09e76283ad3fff3d35d8d0|a21d8cb> means that the build will effectively ignore the exact

complete_platform

JSON files, and use the host system's one. If the host system has newer glibc versions, this might select an incorrect wheel. In this case, the lambda complete platform is CPython 3.9, and thus running with the same Python version hits this issue. Pants version 2.14.0 OS Linux Additional info Complete platform of failing environment as reported by pex: ``` {"path": "/opt/hostedtoolcache/Python/3.9.15/x64/bin/python3.9", "compatible_tags": ["cp39-cp39-manylinux_2_31_x86_64", "cp39-cp39-manylinux_2_30_x86_64", "cp39-cp39-manylinux_2_29_x86_64", "cp39-cp39-manylinux_2_28_x86_64", "cp39-cp39-manylinux_2_27_x86_64", "cp39-cp39-manylinux_2_26_x86_64", "cp39-cp39-manylinux_2_25_x86_64", "cp39-cp39-manylinux_2_24_x86_64", "cp39-cp39-manylinux_2_23_x86_64", "cp39-cp39-manylinux_2_22_x86_64", "cp39-cp39-manylinux_2_21_x86_64", "cp39-cp39-manylinux_2_20_x86_64", "cp39-cp39-manylinux_2_19_x86_64", "cp39-cp39-manylinux_2_18_x86_64", "cp39-cp39-manylinux_2_17_x86_64", "cp39-cp39-manylinux2014_x86_64", "cp39-cp39-manylinux_2_16_x86_64", "cp39-cp39-manylinux_2_15_x86_64", "cp39-cp39-manylinux_2_14_x86_64", "cp39-cp39-manylinux_2_13_x86_64", "cp39-cp39-manylinux_2_12_x86_64", "cp39-cp39-manylinux2010_x86_64", "cp39-cp39-manylinux_2_11_x86_64", "cp39-cp39-manylinux_2_10_x86_64", "cp39-cp39-manylinux_2_9_x86_64", "cp39-cp39-manylinux_2_8_x86_64", "cp39-cp39-manylinux_2_7_x86_64", "cp39-cp39-manylinux_2_6_x86_64", "cp39-cp39-manylinux_2_5_x86_64", "cp39-cp39-manylinux1_x86_64", "cp39-cp39-linux_x86_64", "cp39-abi3-manylinux_2_31_x86_64", "cp39-abi3-manylinux_2_30_x86_64", "cp39-abi3-manylinux_2_29_x86_64", "cp39-abi3-manylinux_2_28_x86_64", "cp39-abi3-manylinux_2_27_x86_64", "cp39-abi3-manylinux_2_26_x86_64", "cp39-abi3-manylinux_2_25_x86_64", "cp39-abi3-manylinux_2_24_x86_64", "cp39-abi3-manylinux_2_23_x86_64", "cp39-abi3-manylinux_2_22_x86_64", "cp39-abi3-manylinux_2_21_x86_64", "cp39-abi3-manylinux_2_20_x86_64", "cp39-abi3-manylinux_2_19_x86_64", "cp39-abi3-manylinux_2_18_x86_64", "cp39-abi3-manylinux_2_17_x86_64", "cp39-abi3-manylinux2014_x86_64", "cp39-abi3-manylinux_2_16_x86_64", "cp39-abi3-manylinux_2_15_x86_64", "cp39-abi3-manylinux_2_14_x86_64", "cp39-abi3-manylinux_2_13_x86_64", "cp39-abi3-manylinux_2_12_x86_64", "cp39-abi3-manylinux2010_x86_64", "cp39-abi3-manylinux_2_11_x86_64", "cp39-abi3-manylinux_2_10_x86_64", "cp39-abi3-manylinux_2_9_x86_64", "cp39-abi3-manylinux_2_8_x86_64", "cp39-abi3-manylinux_2_7_x86_64", "cp39-abi3-manylinux_2_6_x86_64", "cp39-abi3-manylinux_2_5_x86_64", "cp39-abi3-manylinux1_x86_64", "cp39-abi3-linux_x86_64", "cp39-none-manylinux_2_31_x86_64", "cp39-none-manylinux_2_30_x86_64", "cp39-none-manylinux_2_29_x86_64", "cp39-none-manylinux_2_28_x86_64", "cp39-none-manylinux_2_27_x86_64", "cp39-none-manylinux_2_26_x86_64", "cp39-none-manylinux_2_25_x86_64", "cp39-none-manylinux_2_24_x86_64", "cp39-none-manylinux_2_23_x86_64", "cp39-none-manylinux_2_22_x86_64", "cp39-none-manylinux_2_21_x86_64", "cp39-none-manylinux_2_20_x86_64", "cp39-none-manylinux_2_19_x86_64", "cp39-none-manylinux_2_18_x86_64", "cp39-none-manylinux_2_17_x86_64", "cp39-none-manylinux2014_x86_64", "cp39-none-manylinux_2_16_x86_64", "cp39-none-manylinux_2_15_x86_64", "cp39-none-manylinux_2_14_x86_64", "cp39-none-manylinux_2_13_x86_64", "cp39-none-manylinux_2_12_x86_64", "cp39-none-manylinux2010_x86_64", "cp39-none-manylinux_2_11_x86_64", "cp39-none-manylinux_2_10_x86_64", "cp39-none-manylinux_2_9_x86_64", "cp39-none-manylinux_2_8_x86_64", "cp39-none-manylinux_2_7_x86_64", "cp39-none-manylinux_2_6_x86_64", "cp39-none-manylinux_2_5_x86_64", "cp39-none-manylinux1_x86_64", "cp39-none-linux_x86_64", "cp38-abi3-manylinux_2_31_x86_64", "cp38-abi3-manylinux_2_30_x86_64", "cp38-abi3-manylinux_2_29_x86_64", "cp38-abi3-manylinux_2_28_x86_64", "cp38-abi3-manylinux_2_27_x86_64", "cp38-abi3-manylinux_2_26_x86_64", "cp38-abi3-manylinux_2_25_x86_64", "cp38-abi3-manylinux_2_24_x86_64", "cp38-abi3-manylinux_2_23_x86_64", "cp38-abi3-manylinux_2_22_x86_64", "cp38-abi3-manylinux_2_21_x86_64", "cp38-abi3-manylinux_2_20_x86_64", "cp38-abi3-manylinux_2_19_x86_64", "cp38-abi3-manylinux_2_18_x86_64", "cp38-abi3-manylinux_2_17_x86_64", "cp38-abi3-manylinux2014_x86_64", "cp38-abi3-manylinux_2_16_x86_64", "cp38-abi3-manylinux_2_15_x86_64", "cp38-abi3-manylinux_2_14_x86_64", "cp38-abi3-manylinux_2_13_x86_64", "cp38-abi3-manylinux_2_12_x86_64", "cp38-abi3-manylinux2010_x86_64", "cp38-abi3-manylinux_2_11_x86_64", "cp38-abi3-manylinux_2_10_x86_64", "cp38-abi3-manylinux_2_9_x86_64", "cp38-abi3-manylinux_2_8_x86_64", "cp38-abi3-manylinux_2_7_x86_64", "cp38-abi3-manylinux_2_6_x86_64", "cp38-abi3-manylinux_2_5_x86_64", "cp38-abi3-manylinux1_x86_64", "cp38-abi3-linux_x86_64", "cp37-abi3-manylinux_2_31_x86_64", "cp37-abi3-manylinux_2_30_x86_64", "cp37-abi3-manylinux_2_29_x86_64", "cp37-abi3-manylinux_2_28_x86_64", "cp37-abi3-manylinux_2_27_x86_64", "cp37-abi3-manylinux_2_26_x86_… pantsbuild/pants

#17621 python_awslambda can include distributions incompatible with explicit complete_platforms Issue closed as completed by jsirois pantsbuild/pants