https://pantsbuild.org/ logo
Join Slack
Powered by
I just upgraded to 2.26.0 (from 2.24.1) and starte...
# general
g
I just upgraded to 2.26.0 (from 2.24.1) and started seeing this error. I'm curious if anyone knows what the hell is causing this. Seems unrelated to pants, but I also don't know why upgrading pants would start this. Oh, and I also updated pex-cli from v2.20.2 to v2.32.4. > No pre-built wheel was available for pydevd-pycharm 243.22562.220. > > Successfully built the wheel pydevd_pycharm-243.22562.220-cp311-cp311-linux_x86_64.whl from the sdist pydevd_pycharm-243.22562.220.tar.gz but it is not compatible with the requested foreign target complete platform cp39-cp39-manylinux_2_36_x86_64. > > You'll need to build a wheel from pydevd_pycharm-243.22562.220.tar.gz on the foreign target platform and make it available to Pex via a 
--find-links
repo or a custom 
--index
.
h
How is pydevd-pycharm specified in your requirements and lockfile? And what are your interpreter constraints?
g
interpreter constraints for source is 
>3.9.1,<3.11
Copy code
binary_deps_target_name = f"binary-deps-{arch}"
    pex_binary(
        name=binary_deps_target_name,
        include_sources=False,
        include_tools=True,
        venv_hermetic_scripts=False,
        complete_platforms=[
            f"3rdparty/platforms:docker_python_3_9_bookworm_{arch}",
        ],
        dependencies=pex_deps,
        # Optimal settings for Docker builds
        layout="packed",
        execution_mode="venv",
    )
I don't know why pants is building the whl for cp311
h
Do you depend on 
pydevd-pycharm
at all? Is it in your requirements/lockfile? Typically you bring that dep in temporarily when you want to remote-debug with pycharm
Pants does specify it in its own requirements, for that reason, but that shouldn’t be relevant (and anyway, it pins it at a different version than the one you mention above)
g
We are manually adding dependencies to this requirement for this:
Copy code
python_requirement(
    name="debugger",
    requirements=["pydevd-pycharm~=243.22562.220", "debugpy~=1.8.11"],
)
@happy-kitchen-89482 any ideas?
I rolled back the upgrade so no rush, but I would love to resolve the issue.
I also can't replicate it on my mac locally, only on our CI agents running Ubuntu 22.04.
I'm really just trying to get clarity on: Is this a pants bug or an issue with our environment?
h
I would assume a Pants bug. Even if it could be fixed by a change to the environment, that shouldn’t be necessary.
✅ 1
g
Do you need a bug report?
h
Although… can you inspect the contents of your complete platforms file and verify that it is correct
or even regenerate it and compare
g
I can re-gen, sure.
Based on the filename of the build it seems like it's using cp311 for the build and our complete platforms is from cp39
Do you think re-generating with the latest pex could change results?
This is the README.md I created for this >
Copy code
# 3rd party platforms
> 
> This contains platforms used for PEX binaries so we can build for multiple platforms, i.e. `linux/amd64`, `linux/arm64`
> 
> ## Create Platform Target(s)
> 
> In this example I am creating targets for 3.9 bookworm
> 
> 1. Generate JSON files
> 
>    ```sh
>    docker run -it --platform linux/amd64 public.ecr.aws/docker/library/python:3.9-slim-bookworm bash -c 'pip install pex; pex3 interpreter inspect --markers --tags' | grep platform_python_implementation | jq . >| docker_python_3_9_bookworm_amd64.json
>
> >
Copy code
sh
>    docker run -it --platform linux/arm64 public.ecr.aws/docker/library/python:3.9-slim-bookworm bash -c 'pip install pex; pex3 interpreter inspect --markers --tags' | grep platform_python_implementation | jq . >| docker_python_3_9_bookworm_arm64.json
>
> > 2. Create new files target > >
Copy code
text
>    files(
>        name="docker_python_3_9_bookworm",
>        sources=["docker_python_3_9_bookworm_*.json"],
>    )
>
> > 3. Reference target in `pex_binary`'s 
complete_platforms
argument > >
Copy code
text
>    pex_binary(
>        name="pex_binary",
>        complete_platforms=[
>            "3rdparty/platforms:docker_python_3_9_bookworm",
>        ],
>    )
>    ```
platform diff.diff
platform diff.diff
When I re-generated, this was the only delta. ^^
I kicked off a new build in CI to see if the re-generated platforms files fixes it.
should know in a few minutes.
Yeah, same error.
h
OK, it was worth checking
So, yeah, a repro would be great
Or, try reproducing this outside Pants, by running with 
--keep-sandboxes=on_failure
and looking inside the sandbox (whose location will be logged to the console)
Specifically, at 
__run.sh
g
yeah the issue is that it's using the venv from the pants scie, I think.
Copy code
/nvme/home/agent-13/.cache/nce/6d61748cea187199dc28418157b2121ccbbb44e4af2ee6fddf8c04bbed73f76d/bindings/venvs/2.26.0/bin/python3.11
and in this case it should be using a 3.9 interpreter
oh I don't know what I'm talking about...
It's passing in the complete platforms.
in any case, I can reproduce when executing __run.sh
full command in `__run.sh`:
Copy code
env -i CPPFLAGS='' LANG=C.UTF-8 LDFLAGS='' PATH=$'/usr/local/go/bin:/nvme/home/agent-13/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/nvme/home/agent-13/.cache/nce/6d61748cea187199dc28418157b2121ccbbb44e4af2ee6fddf8c04bbed73f76d/bindings/venvs/2.26.0/bin/python3.11 PEX_ROOT=.cache/pex_root /nvme/home/agent-13/.cache/nce/6d61748cea187199dc28418157b2121ccbbb44e4af2ee6fddf8c04bbed73f76d/bindings/venvs/2.26.0/bin/python3.11 ./pex --tmpdir .tmp --jobs 32 --no-emit-warnings --pip-version 24.2 --python-path $'/usr/local/go/bin:/nvme/home/agent-13/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin' --output-file apps.atlas-api/binary-deps-amd64.pex --emit-warnings $'--check=warn' --venv prepend --include-tools --non-hermetic-venv-scripts --requirements-pex local_dists.pex --complete-platform 3rdparty/platforms/docker_python_3_9_bookworm_amd64.json $'--sources-directory=source_files' $'SQLAlchemy<2.0.0,>=1.4' $'aiohttp<4.0.0,>=3.11.2' $'aiohttp<4.0.0,>=3.8.4' $'alembic<2.0.0,>=1.8.0' $'authlib<2.0.0,>=1.2' $'azure-core<2.0.0,>=1.25.0' azure-identity azure-servicebus $'azure-storage-blob<13.0.0,>=12.16.0' $'bleach<7.0.0,>=6.0.0' $'boto3<2.0.0,>=1.35.16' $'boto3<2.0.0,>=1.35.99' $'boto3==1.35.99' $'botocore<2.0.0,>=1.35.99' $'cachetools<6.0.0,>=5.3.2' $'celery[redis]<6.0.0,>=5.4.0' $'connexion[swagger-ui]<3.0.0,>=2.14' $'country-converter<2.0.0,>=1.0' $'cryptography<45.0.0,>=42.0.5' $'databricks-cli==0.18.0' $'databricks-sql-connector<3.0.0,>=2.0.5' $'databricks-sql-connector>=2' $'datadog<1.0.0,>=0.48.0' $'ddtrace<3.0.0,>=2' $'debugpy~=1.8.11' $'docker<8.0.0,>=7.1.0' $'editdistance<0.7.0,>=0.6.2' $'flask-compress<2.0.0,>=1.12' $'flask-cors<5.0.0,>=4.0.2' $'flask==2.2.5' $'flask_testing<0.9.0,>=0.8' $'flower<2.0.0,>=1.2.0' $'fpdf2<3.0.0,>=2.8.2' $'gunicorn<23.0.0,>=22.0' $'hvac<0.11.0,>=0.10' $'inflect<7.0.0,>=6.0.4' $'jsonschema<5.0.0,>=4.10' $'mergedeep<2.0.0,>=1.3.4' $'moto[s3,sqs]<5.0,>=3.0' $'mypy-boto3-quicksight<2.0.0,>=1.28.16' $'mypy-boto3-s3<2.0.0,>=1.26.58' $'mypy-boto3-ses<2.0.0,>=1.34.141' $'mypy-boto3-sts<2.0.0,>=1.26.58' $'networkx<4.0.0,>=3' $'nltk<4.0.0,>=3.8.1' $'numpy<2,>=1.21.5' $'opensearch-py<3.0.0,>=2.3.2' $'opensearch-py<3.0.0,>=2.6.0' $'opensearch-py[async]<3.0.0,>=2.4.2' $'orjson<3.10.0,>=3.9.15' $'pandas-stubs>=1.5.0.221010' $'pandas<2,>=1' $'pgvector<0.3.0,>=0.2.1' $'psycopg2-binary<3.0.0,>=2.9.3' $'py-healthcheck<2.0.0,>=1.10.1' $'pyarrow-hotfix<0.7.0,>=0.6' $'pyarrow>=9.0.0' $'pydantic-settings==2.2.1' $'pydantic==2.7.1' $'pydevd-pycharm~=243.22562.220' $'pyjwt<3.0.0,>=2.8.0' $'pytest<=8.3.2,>=7.2.1' $'python-arango<8.0.0,>=7.1' $'python-arango==7.5.7' $'python-json-logger<3.0.0,>=2.0.4' $'python_dateutil<3.0.0,>=2.6' $'pyyaml<7.0.0,>=6' $'redis[hiredis]<4.7.0,>=4.6' $'requests<2.32,>=2' $'requests<3.0.0,>=2.31.0' $'rich<13.0.0,>=12.6.0' $'six<2.0.0,>=1.16' $'sqlalchemy-stubs<0.4.0,>=0.3' $'tenacity>=8.2' $'testcontainers<5.0.0,>=4.10.0' $'types-bleach<7.0.0,>=6.0.0.3' $'types-cachetools<6.0.0,>=5.3.0.5' $'types-cachetools<6.0.0,>=5.3.0.7' $'types-python-dateutil==2.8.14' $'types-pytz<2023.0.0,>=2022.5.0.0' $'types-pyyaml<7.0.0,>=6' $'types-redis<5.0.0,>=4.6.0.5' $'types-requests<2.32,>=2' $'types-requests<3.0.0,>=2.30.0.0' $'types-requests<3.0.0,>=2.31.0.0' $'types-six<2.0.0,>=1.16.21' $'typing-extensions<5.0.0,>=4' $'unidecode<2.0.0,>=1.3.6' $'urllib3<2.0.0,>=1.26.9' $'werkzeug==2.2.3' --lock 3rdparty/python/web.lock --manylinux manylinux2014 --layout packed
docker_python_3_9_bookworm_amd64.json
docker_python_3_9_bookworm_amd64.json
The only delta between pants 2.24.1 and 2.26.0 in the generated 
__run.sh
is the python interpreter. So I don't know if it was just "luck" that the interpreter of pants aligned with our interpreter constraints and in 2.26.0 it doesn't or if it's something else.
Untitled.diff
Untitled.diff
Setting 
PEX_PYTHON=/usr/bin/python3.9
in 
__run.sh
fixes it.
I don't know if this is a config issue with our pex_binary target or the resolve or what
With both pants 2.24.1 and 2.26.0 I see the exact same log in terms of interpreter being resolved:
Copy code
cp310-cp310-manylinux_2_35_x86_64 interpreter at /usr/bin/python3.10
cp39-cp39-manylinux_2_35_x86_64 interpreter at /usr/bin/python3.9
Using shebang: #!/usr/bin/env python3.9
02:30:49.85 [DEBUG] Completed: Find Python interpreter for constraints - environment:linux - Selected /usr/bin/python3.9 to run PEXes with.
Let me know if there is any other digging I can do.
Yeah, so it's selected the right python interpreter, but somehow that's not getting plugged into PEX_PYTHON.
h
Are you able to create a small repo that reproduces this?
g
Probably with enough time.
hope that helps!
I couldn't repro it with just the debugger so I added a ton of deps which got it to trigger.
h
Thanks! Sounds like we’ll also need to know the image this reproduces in (as it may depend on the presence of specific python interpreters)
g
I can reproduce on macOS 15.4.1 and Ubuntu 22.04. Were you unable to reproduce?
h
Oh I thought you said you couldn’t reproduce on macos
Ah, I see in the README that on macos you get a similar error, related to thrift. But it’s not surprising that you can’t build an sdist into a linux wheel on macos, so this may be a red herring.
But anyway, I’ll take a look
g
It's the exact same error, just a different package. Thanks
I realized I couldn't re-produce on macOS originally because we are using environments and on macOS we don't match the compatible_platforms, so we build in a container. I can bypass/workaround this issue if I use a container for the build on both Linux and macOS.
h
It is not quite the same - naturally you can’t build an sdist into a linux wheel on native macos, so it’s not surprising that this fails.
There is the question of why pex even tries to build the python3.11 macos wheel in this case though. I’m not sure about that, will need to remind myself how things work here.
That said, can you try regenerating the complete_platforms file on the CI image? Any incompatibility between the platform you generated that file on and the platform you try and use it on will cause issues like this
So I don’t know if that image you’re generating complete_platforms on is appropriate
g
Ah I see. I can do that on Monday and let you know.
We have 4 python interpreters on our CI agents
h
It’s more about platform tag compatibility, probably
But of course the question is why it would fail on upgrade, since this compatibility requirement has always been there
FWIW - I could not reproduce this on an ubuntu 22.04 AMI with python3.9 installed via deadsnakes
I can reproduce the similar issue on macos, but that is (as mentioned) not surprising
Although I will check to see if that would have happened with Pants 2.24.1 too
Yep, it does happen on macos on 2.24.1, as expected.
So yeah, to reproduce the unexpected issue, will need to know exactly which image to try this on (and I recommend checking the complete_platforms generated on that image vs the one you use)
g
Did you see the comments on the repo by @brief-scientist-13682 ?
It looks like this is completely expected and we just got lucky by the fact that our version of python matched that of pants.
I’ll probably end up using environments with docker to resolve this unless you think pants should do something here to help out.
Or build wheels for the platforms and publish to an internal pypi repository
h
Yep, so what he calls "YOLO mode" is the answer to my question above: "There is the question of why pex even tries to build the python3.11 macos wheel in this case though."
But beyond that the failure on macos is, as mentioned here and by him there, unsurprising
And the failure on Linux is slightly more subtle, and naively more surprising, but expected nonetheless
That one could be solved by running pex itself on the target python
Hence the subtlety
g
For transparency. I fixed this issue by building wheels for the 3rd party packages that only distribute sdists, i.e. thrift and pydevd-pycharm I'm on 2.26 now! 🎉 Thanks @brief-scientist-13682
37 Views
Open in Slack
PreviousNext
Powered by