Howdy I m trying to run pants 2 8 0 with `interpreter constr Pants #general

Howdy. I'm trying to run pants 2.8.0 with `interpr...

billions-spring-9652

12/19/2021, 4:44 PM

Howdy. I'm trying to run pants 2.8.0 with

interpreter_constraints = ["CPython==3.10.*"]

. I'm on ubuntu 18.04 (bionic) and my python3.10 binary is from the deadsnakes ppa. This is what I get:

ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython==3.10.*' failed with exit code 102.

stdout:

stderr:

Could not find a compatible interpreter.

Examined the following interpreters:

1.) /usr/bin/python2.7 CPython==2.7.17

2.) /usr/bin/python3.6 CPython==3.6.9

3.) /usr/bin/python3.8 CPython==3.8.0

4.) /usr/bin/python3.9 CPython==3.9.9

(That 3.9.9 version is from deadsnakes as well. If I specify CPython==3.9.*, it's picked up and everything's fine). So, is my python3.10 installation broken somehow, or not supported?

fast-nail-55400

12/19/2021, 4:52 PM

What is the path to that 3.10 interpreter? And what is the search path set to in

pants.toml

? From that output, it seems like Pants did not even find the

python

binary.

billions-spring-9652

12/19/2021, 4:56 PM

/usr/bin/python3.10, and the current pants.toml is a stripped down one that doesn't specify interpreter_search_paths. I had one that specified

["<PYENV>", "/usr/local/bin", "/usr/bin"]

earlier. Same effect.

enough-analyst-54434

12/19/2021, 5:02 PM

Pants 2.8.0 uses Pex 2.1.54 (2.8.0rc2 had the upgrade cherry pick). Pex 2.1.55 is the first release with official Python 3.10 support: https://github.com/pantsbuild/pex/blob/main/CHANGES.rst#2155 I'm surprised Pex 2.1.54 is failing to identify a python 3.10 interpreter, but even if that worked, later use to build wheels from sdists would have failed anyhow (This vendored setuptools patch was needed: https://github.com/pantsbuild/setuptools/pull/1).

enough-analyst-54434

12/19/2021, 5:04 PM

So, short answer, is this is a no go on 2.8.0. I can work on a Pex upgrade cherry-pick though for 2.8.1rc2 though later today if this is critical to have before 2.9.0 (we just released 2.9.0rc0).

billions-spring-9652

12/19/2021, 5:10 PM

An rc is ok for me. But I just changed to

pants_version = "2.9.0rc0"

, and I get the same thing. I suppose there's something specific going on with my setup.

billions-spring-9652

12/19/2021, 5:12 PM

These deadsnakes python versions are newly installed, and haven't seen any real use yet. This is a build server with lots of stuff installed, so there are opportunities for things to go wrong.

billions-spring-9652

12/19/2021, 5:44 PM

Oh. I'm playing around in the process-execution directory, and I've discovered something interesting.

python3 ./pex --interpreter-constraint $'CPython==3.10.*' -- -c 'print(42)'

gives the familiar constraint failure, but if I specifically invoke python3.10, that is:

python3.10 ./pex --interpreter-constraint $'CPython==3.10.*' -- -c 'print(42)'

, that works!

enough-analyst-54434

12/19/2021, 6:20 PM

Do the dead snakes pythons rely on a custom LD_LIBRARY_PATH per chance?

enough-analyst-54434

12/19/2021, 6:24 PM

Likely more on point, what exact packages do you install from deadsnakes? Reading here: https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa it looks like Debian-style "many packages composing a full install" is maintained. You may need the -distutils and -venv packages too.

enough-analyst-54434

12/19/2021, 6:27 PM

See: https://github.com/pantsbuild/pex/issues/1027

enough-analyst-54434

12/19/2021, 8:07 PM

I'll see if I can repro this in a Bionic container here in a bit.

billions-spring-9652

12/19/2021, 8:07 PM

I don't believe there's any LD_LIBRARY_PATH trickery going on with the deadsnakes packages. The packages I installed initially are python3.10 and python3.10-venv (which also pulls in python3.10-distutils). I didn't have python3-distlib initially, but now I do.

billions-spring-9652

12/19/2021, 8:26 PM

I've started on a Dockerfile to reproduce this myself, so you may want to hold off on that.

billions-spring-9652

12/19/2021, 9:06 PM

All right. So the official ubuntu:18.04 image has the same problem.

pants.toml Dockerfile

enough-analyst-54434

12/19/2021, 9:06 PM

I repro:

Copy code

$ cat Dockerfile 
FROM ubuntu:18.04

RUN apt-get update && \
    apt-get install -y software-properties-common && \
    add-apt-repository -y ppa:deadsnakes/ppa

RUN apt-get update && \
    apt-get install -y \
    curl \
    python \
    python-dev \
    python3.8 \
    python3.8-dev \
    python3.8-venv \
    python3.8-distutils \
    python3.9 \
    python3.9-dev \
    python3.9-venv \
    python3.9-distutils \
    python3.10 \
    python3.10-dev \
    python3.10-venv \
    python3.10-distutils

$ docker build -t bionic:deadsnakes-310 .
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM ubuntu:18.04
 ---> 5a214d77f5d7
Step 2/3 : RUN apt-get update &&     apt-get install -y software-properties-common &&     add-apt-repository -y ppa:deadsnakes/ppa
 ---> Using cache
 ---> b1b5a578052e
Step 3/3 : RUN apt-get update &&     apt-get install -y     curl     python     python-dev     python3.8     python3.8-dev     python3.8-venv     python3.8-distutils     python3.9     python3.9-dev     python3.9-venv     python3.9-distutils     python3.10     python3.10-dev     python3.10-venv     python3.10-distutils
 ---> Using cache
 ---> 29464760100f
Successfully built 29464760100f
Successfully tagged bionic:deadsnakes-310
$ docker run --rm -it bionic:deadsnakes-310 bash -i
root@f4f52262c693:/# curl -sSL <https://github.com/pantsbuild/pex/releases/download/v2.1.56/pex> > pex-2.1.56 
root@f4f52262c693:/# chmod +x pex-2.1.56 
root@f4f52262c693:/# ./pex-2.1.56 --interpreter-constraint "==3.11.*"         
Could not find a compatible interpreter.

Examined the following interpreters:
1.) /usr/bin/python2.7 CPython==2.7.17
2.) /usr/bin/python3.6 CPython==3.6.9
3.) /usr/bin/python3.8 CPython==3.8.12
4.) /usr/bin/python3.9 CPython==3.9.9

No interpreter compatible with the requested constraints was found:
  Version matches ==3.11.*
root@f4f52262c693:/#

enough-analyst-54434

12/19/2021, 9:07 PM

I'll dig here a bit. Thanks for the data @billions-spring-9652.

enough-analyst-54434

12/19/2021, 9:10 PM

Yeah - somehow Pex is not seeing that interpreter on the $PATH even though it is:

Copy code

root@f4f52262c693:/# rm -rf ~/.pex/
root@f4f52262c693:/# ./pex-2.1.56 -vvvvvvvvvv --interpreter-constraint "==3.11.*"
pex: Spawning a maximum of 16 parallel jobs to process:
  /usr/bin/python
  /usr/bin/python2
  /usr/bin/python2.7
  /usr/bin/python3
  /usr/bin/python3.6
  /usr/bin/python3.6m
  /usr/bin/python3.8
  /usr/bin/python3.9
pex: Hashing pex: 20.0ms
...

billions-spring-9652

12/19/2021, 9:11 PM

You've got 3.11 in there though.

enough-analyst-54434

12/19/2021, 9:11 PM

Yeah - I was smoking out the full list by asking Pex for something I know doesn;t exist.

billions-spring-9652

12/19/2021, 9:12 PM

ah, I see

enough-analyst-54434

12/19/2021, 9:12 PM

Pex finds 3.10 in GH actions CI ... so this will be interesting.

billions-spring-9652

12/19/2021, 9:15 PM

It'll be one of those debian python packaging weirdnesses, I guess.

enough-analyst-54434

12/19/2021, 9:15 PM

We'll see. Follow along here: https://github.com/pantsbuild/pex/issues/1544

billions-spring-9652

12/19/2021, 9:18 PM

All right. Thanks for helping me out.

enough-analyst-54434

12/19/2021, 9:23 PM

It ... its embarrassing. A fix for this will make it into 2.9.0rc1.

billions-spring-9652

12/19/2021, 9:32 PM

Hehe, ok 🙂

8 Views

Open in Slack

Previous Next