https://pantsbuild.org/ logo
#general
Title
# general
f

fresh-cat-90827

01/11/2022, 7:47 PM
I saw an error message today when bootstrapping Pants in CI (Linux), but can’t reproduce locally (the error is gone for a fresh build of a Git checkout now). Thought to share anyway in case it’s a symptom of something important:
Copy code
$ ./pants
Bootstrapping Pants using /usr/bin/python3.8
Creating the virtualenv PEX.
Downloading the Pex PEX.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   640  100   640    0     0   3076      0 --:--:-- --:--:-- --:--:--  3076

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3529k  100 3529k    0     0  5719k      0 --:--:-- --:--:-- --:--:-- 17.2M

SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.42/pex> verified.
Traceback (most recent call last):
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 484, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 540, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 656, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
    runner = entry.resolve()
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/bin/pex.py", line 21, in <module>
    from pex.interpreter import PythonInterpreter
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/interpreter.py", line 25, in <module>
    from pex.platforms import Platform
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/platforms.py", line 13, in <module>
    from pex.third_party.packaging import tags
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/vendor/_vendored/packaging/packaging/tags.py", line 7, in <module>
    import distutils.util

  File "/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py", line 92, in create_module
    return importlib.import_module('setuptools._distutils')
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools'
mv: cannot stat '/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/pants.hxhmpE/virtualenv.pex': No such file or directory
Installing pantsbuild.pants==2.7.1 into a virtual environment at /home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.1_py38
/usr/bin/python3.8: can't open file '/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pex': [Errno 2] No such file or directory
script returned exit code 1
Just to clarify, starting subsequent builds on the failed build (while keeping the current build workspace) were failing as well, but the fresh build with a new Git checkout into a new build workspace worked
f

fast-nail-55400

01/11/2022, 7:50 PM
ModuleNotFoundError: No module named 'setuptools'
seems like the error which would seem to imply that maybe some module is not declaring a dependency on setuptools
although this is the setup of the venv for pants itself?
f

fresh-cat-90827

01/11/2022, 7:54 PM
thanks @fast-nail-55400, exactly, Pants didn’t yet look at any sources, it’s just being bootstrapped 🙂
I’ve modified the error output to indicate the command that was run —
./pants
f

fast-nail-55400

01/11/2022, 7:59 PM
does the error reproduce if you run
./pants
manually in the CI environment? also, what Python installation is Pants using on the CI machine? what distribution of Linux is it?
f

fresh-cat-90827

01/11/2022, 8:06 PM
does the error reproduce if you run 
./pants
 manually in the CI environment?
It’s a Dockerized build environment on a remote server, so I can’t just run some arbitrary commands there 🙂 It’s
ubuntu:focal
Docker image with
python3.8
system package.
f

fast-nail-55400

01/11/2022, 10:44 PM
ubuntu splits off setuptools and distutils into separate debs
are
python3-venv
and `python3-distutils`packages installed in the image?
1
f

fresh-cat-90827

01/17/2022, 3:36 PM
After dozens of successful builds, the problem manifests itself again with identical error message:
ModuleNotFoundError: No module named 'setuptools'
.
are python3-venv and python3-distutilspackages installed in the image?
Yes, I have the following system packages in the Docker image used in CI installed.
Copy code
python3.8-distutils \
       python3.8-venv \
@enough-analyst-54434 I am sorry, could you confirm that this has nothing to do with the
pex
itself, when you get a chance, please? I can’t comprehend how in one run using the same Dockerfile the
setuptools
is found and another one isn’t. Doesn’t make much sense. 😕 I am on
2.7.1
e

enough-analyst-54434

01/17/2022, 9:34 PM
The file "/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py" in your backtrace is from setuptools and was introduced in setuptools 49.3.0 (https://github.com/pypa/setuptools/commit/dcc71f773576c19a3658735879893515b056ece5#diff-f89a3e6e15487ff639f3087ce7[…]918466ee6e7d5bf62c9ae648445cad8). Pex vendors setuptools 44.0.0; so this means a system (`apt install`ed) setuptools is tripping the PEX up. Let me see if I can concoct a repro via Docker.
@fresh-cat-90827 any chance you can provide me with your Dockerfile(s) or the relevant bits?
It would be good to know where the system setuptools is coming from / what version it is for repro purposes.
Yeah, using `ubuntu:focal`with
apt update && apt install -y python3.8-distutils python3.8-venv python3-setuptools
I find:
Copy code
root@4002c0b0e2e1:/# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
root@4002c0b0e2e1:/# python3.8 -c 'import sys; print("\n".join(sys.path))'

/usr/lib/python38.zip
/usr/lib/python3.8
/usr/lib/python3.8/lib-dynload
/usr/local/lib/python3.8/dist-packages
/usr/lib/python3/dist-packages
root@4002c0b0e2e1:/# ls -l /usr/local/lib/python3.8/dist-packages
total 0
root@4002c0b0e2e1:/# ls -l /usr/lib/python3/dist-packages
total 20
drwxr-xr-x 2 root root 4096 Jan 17 21:53 __pycache__
-rw-r--r-- 1 root root  126 Feb  9  2020 easy_install.py
drwxr-xr-x 5 root root 4096 Jan 17 21:47 pkg_resources
drwxr-xr-x 6 root root 4096 Jan 17 21:53 setuptools
drwxr-xr-x 2 root root 4096 Jan 17 21:53 setuptools-45.2.0.egg-info
root@4002c0b0e2e1:/#
So that setuptools (which is the only way I could see to get it for system python 3.8) is both not the same version as yours (its lower) and its not installed in the same sys.path entry.
@fresh-cat-90827 so your repro data is pretty critical here when you get a chance to provide it.
f

fresh-cat-90827

01/17/2022, 10:03 PM
That’s so helpful, thanks, John! The Dockerfile is very simple:
Copy code
FROM ubuntu:focal
LABEL name="ci/focal"

# Install build tools
RUN DEBIAN_FRONTEND=noninteractive apt-get -qq update \
    && apt-get install -y --no-install-recommends \
        apt-transport-https \
        ca-certificates \
...
       python3.8 \
       python3.8-dev \
       python3.8-distutils \
       python3.8-venv \
...

RUN wget -q <https://bootstrap.pypa.io/get-pip.py> \
       && python3 get-pip.py pip==21.0.1 \
       && rm get-pip.py
...
The relevant bits of
Dockerfile
are above. Some system info:
Copy code
$ python3 --version
Python 3.8.10
Python system packages:
Copy code
$ apt list --installed | grep python

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libpython3-stdlib/focal,now 3.8.2-0ubuntu2 amd64 [installed,automatic]
libpython3.8-dev/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
libpython3.8-minimal/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
libpython3.8-stdlib/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
libpython3.8/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
python-apt-common/focal-updates,now 2.0.0ubuntu0.20.04.6 all [installed,automatic]
python-pip-whl/focal-updates,now 20.0.2-5ubuntu1.6 all [installed,automatic]
python3-apt/focal-updates,now 2.0.0ubuntu0.20.04.6 amd64 [installed,automatic]
python3-certifi/focal,now 2019.11.28-1 all [installed,automatic]
python3-chardet/focal,now 3.0.4-4build1 all [installed,automatic]
python3-dbus/focal,now 1.2.16-1build1 amd64 [installed,automatic]
python3-distutils/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04 all [installed]
python3-gi/focal,now 3.36.0-1 amd64 [installed,automatic]
python3-idna/focal,now 2.8-1 all [installed,automatic]
python3-lib2to3/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04 all [installed,automatic]
python3-minimal/focal,now 3.8.2-0ubuntu2 amd64 [installed,automatic]
python3-pkg-resources/focal,now 45.2.0-1 all [installed,automatic]
python3-requests-unixsocket/focal,now 0.2.0-2 all [installed,automatic]
python3-requests/focal,now 2.22.0-2ubuntu1 all [installed,automatic]
python3-six/focal,now 1.14.0-2 all [installed,automatic]
python3-software-properties/focal-updates,now 0.99.9.8 all [installed,automatic]
python3-urllib3/focal-updates,focal-security,now 1.25.8-2ubuntu0.1 all [installed,automatic]
python3.8-dev/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed]
python3.8-minimal/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
python3.8-venv/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed]
python3.8/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed]
python3/focal,now 3.8.2-0ubuntu2 amd64 [installed,automatic]
And setuptools:
Copy code
$ /usr/bin/python3 -c "import setuptools; print(setuptools.__version__)"
60.5.0
Let me know if I’ve missed anything.
Oh you see, I don’t install in the Dockerfile at all
Copy code
python3-setuptools
whereas you do
apt update && apt install -y python3.8-distutils python3.8-venv python3-setuptools
e

enough-analyst-54434

01/17/2022, 10:06 PM
Its the get-pip curl that does it.
👀 1
Thanks.
Ok, I don't repro, but you also said this only happens some of the time - somehow. Is that right?
f

fresh-cat-90827

01/17/2022, 10:17 PM
exactly, that’s the weirdest part. I’ve only seen it two times over the thousands of builds we’ve had so far. The first one happened a couple of weeks ago, and today I’ve got myself the second time.
https://bootstrap.pypa.io/get-pip.py seems to be getting the latest
setuptools
. But maybe due to some other process/tool, a different version of
setuptools
may be installed each time a Docker image is built?
I’ll run the build/run/print version of setuptools loop tomorrow over a few hours to check the hypothesis of mine
e

enough-analyst-54434

01/17/2022, 10:21 PM
Not sure. Either way, Pex tries very hard to guard itself from all system packages. It attempts to scrub them all from the sys.path / sys.modules, etc.
I played with
SETUPTOOLS_USE_DISTUTILS
and used
local
and
stdlib
and neither triggered the issue. Thats a magic env var that has to do with the setuptools
_distutils_hack
machinery.
Ok, thanks @fresh-cat-90827 more data like that will be useful / critical here.
f

fresh-cat-90827

01/17/2022, 10:34 PM
great, give me a few days please, I’ll post later once I have experimented enough. Forever grateful for all the support 🙂
❤️ 1
hey @enough-analyst-54434! So I couldn’t reproduce it locally — it must have been a good chance that in the past the exact same
apt-get install
command in the
Dockerfile
resulted in
python3-setuptools
being installed, but then later it was not the case. Our CI would try to build it (the Docker image) on the same agent it built before, and if the
Dockerfile
has not changed, it will be cached and thus broken until it’s fixed. So I think we were just playing Docker cache lottery. I’ve added
python3-setuptools \
to this list of Python related packages and the problem has gone away since then.
Copy code
python3.8-dev \
       python3.8-distutils \
       python3.8-venv \
3 Views