https://pantsbuild.org/ logo
Join Slack
Powered by
I saw an error message today when bootstrapping Pa...
# general
f
I saw an error message today when bootstrapping Pants in CI (Linux), but can’t reproduce locally (the error is gone for a fresh build of a Git checkout now). Thought to share anyway in case it’s a symptom of something important:
Copy code
$ ./pants
Bootstrapping Pants using /usr/bin/python3.8
Creating the virtualenv PEX.
Downloading the Pex PEX.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   640  100   640    0     0   3076      0 --:--:-- --:--:-- --:--:--  3076

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3529k  100 3529k    0     0  5719k      0 --:--:-- --:--:-- --:--:-- 17.2M

SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.42/pex> verified.
Traceback (most recent call last):
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 484, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 540, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 656, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
    runner = entry.resolve()
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/bin/pex.py", line 21, in <module>
    from pex.interpreter import PythonInterpreter
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/interpreter.py", line 25, in <module>
    from pex.platforms import Platform
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/platforms.py", line 13, in <module>
    from pex.third_party.packaging import tags
  File "/home/jenkins/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/vendor/_vendored/packaging/packaging/tags.py", line 7, in <module>
    import distutils.util

  File "/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py", line 92, in create_module
    return importlib.import_module('setuptools._distutils')
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'setuptools'
mv: cannot stat '/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/pants.hxhmpE/virtualenv.pex': No such file or directory
Installing pantsbuild.pants==2.7.1 into a virtual environment at /home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.1_py38
/usr/bin/python3.8: can't open file '/home/jenkins/.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pex': [Errno 2] No such file or directory
script returned exit code 1
Just to clarify, starting subsequent builds on the failed build (while keeping the current build workspace) were failing as well, but the fresh build with a new Git checkout into a new build workspace worked
f
ModuleNotFoundError: No module named 'setuptools'
seems like the error which would seem to imply that maybe some module is not declaring a dependency on setuptools
although this is the setup of the venv for pants itself?
f
thanks @fast-nail-55400, exactly, Pants didn’t yet look at any sources, it’s just being bootstrapped 🙂
I’ve modified the error output to indicate the command that was run — 
./pants
f
does the error reproduce if you run 
./pants
manually in the CI environment? also, what Python installation is Pants using on the CI machine? what distribution of Linux is it?
f
does the error reproduce if you run 
./pants
 manually in the CI environment?
It’s a Dockerized build environment on a remote server, so I can’t just run some arbitrary commands there 🙂 It’s 
ubuntu:focal
Docker image with 
python3.8
system package.
f
ubuntu splits off setuptools and distutils into separate debs
are 
python3-venv
and `python3-distutils`packages installed in the image?
➕ 1
f
After dozens of successful builds, the problem manifests itself again with identical error message: 
ModuleNotFoundError: No module named 'setuptools'
.
are python3-venv and python3-distutilspackages installed in the image?
Yes, I have the following system packages in the Docker image used in CI installed.
Copy code
python3.8-distutils \
       python3.8-venv \
@enough-analyst-54434 I am sorry, could you confirm that this has nothing to do with the 
pex
itself, when you get a chance, please? I can’t comprehend how in one run using the same Dockerfile the 
setuptools
is found and another one isn’t. Doesn’t make much sense. 😕 I am on 
2.7.1
e
The file "/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py" in your backtrace is from setuptools and was introduced in setuptools 49.3.0 (https://github.com/pypa/setuptools/commit/dcc71f773576c19a3658735879893515b056ece5#diff-f89a3e6e15487ff639f3087ce7[…]918466ee6e7d5bf62c9ae648445cad8). Pex vendors setuptools 44.0.0; so this means a system (`apt install`ed) setuptools is tripping the PEX up. Let me see if I can concoct a repro via Docker.
@fresh-cat-90827 any chance you can provide me with your Dockerfile(s) or the relevant bits?
It would be good to know where the system setuptools is coming from / what version it is for repro purposes.
Yeah, using `ubuntu:focal`with 
apt update && apt install -y python3.8-distutils python3.8-venv python3-setuptools
I find:
Copy code
root@4002c0b0e2e1:/# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
root@4002c0b0e2e1:/# python3.8 -c 'import sys; print("\n".join(sys.path))'

/usr/lib/python38.zip
/usr/lib/python3.8
/usr/lib/python3.8/lib-dynload
/usr/local/lib/python3.8/dist-packages
/usr/lib/python3/dist-packages
root@4002c0b0e2e1:/# ls -l /usr/local/lib/python3.8/dist-packages
total 0
root@4002c0b0e2e1:/# ls -l /usr/lib/python3/dist-packages
total 20
drwxr-xr-x 2 root root 4096 Jan 17 21:53 __pycache__
-rw-r--r-- 1 root root  126 Feb  9  2020 easy_install.py
drwxr-xr-x 5 root root 4096 Jan 17 21:47 pkg_resources
drwxr-xr-x 6 root root 4096 Jan 17 21:53 setuptools
drwxr-xr-x 2 root root 4096 Jan 17 21:53 setuptools-45.2.0.egg-info
root@4002c0b0e2e1:/#
So that setuptools (which is the only way I could see to get it for system python 3.8) is both not the same version as yours (its lower) and its not installed in the same sys.path entry.
@fresh-cat-90827 so your repro data is pretty critical here when you get a chance to provide it.
f
That’s so helpful, thanks, John! The Dockerfile is very simple:
Copy code
FROM ubuntu:focal
LABEL name="ci/focal"

# Install build tools
RUN DEBIAN_FRONTEND=noninteractive apt-get -qq update \
    && apt-get install -y --no-install-recommends \
        apt-transport-https \
        ca-certificates \
...
       python3.8 \
       python3.8-dev \
       python3.8-distutils \
       python3.8-venv \
...

RUN wget -q <https://bootstrap.pypa.io/get-pip.py> \
       && python3 get-pip.py pip==21.0.1 \
       && rm get-pip.py
...
The relevant bits of 
Dockerfile
are above. Some system info:
Copy code
$ python3 --version
Python 3.8.10
Python system packages:
Copy code
$ apt list --installed | grep python

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libpython3-stdlib/focal,now 3.8.2-0ubuntu2 amd64 [installed,automatic]
libpython3.8-dev/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
libpython3.8-minimal/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
libpython3.8-stdlib/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
libpython3.8/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
python-apt-common/focal-updates,now 2.0.0ubuntu0.20.04.6 all [installed,automatic]
python-pip-whl/focal-updates,now 20.0.2-5ubuntu1.6 all [installed,automatic]
python3-apt/focal-updates,now 2.0.0ubuntu0.20.04.6 amd64 [installed,automatic]
python3-certifi/focal,now 2019.11.28-1 all [installed,automatic]
python3-chardet/focal,now 3.0.4-4build1 all [installed,automatic]
python3-dbus/focal,now 1.2.16-1build1 amd64 [installed,automatic]
python3-distutils/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04 all [installed]
python3-gi/focal,now 3.36.0-1 amd64 [installed,automatic]
python3-idna/focal,now 2.8-1 all [installed,automatic]
python3-lib2to3/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04 all [installed,automatic]
python3-minimal/focal,now 3.8.2-0ubuntu2 amd64 [installed,automatic]
python3-pkg-resources/focal,now 45.2.0-1 all [installed,automatic]
python3-requests-unixsocket/focal,now 0.2.0-2 all [installed,automatic]
python3-requests/focal,now 2.22.0-2ubuntu1 all [installed,automatic]
python3-six/focal,now 1.14.0-2 all [installed,automatic]
python3-software-properties/focal-updates,now 0.99.9.8 all [installed,automatic]
python3-urllib3/focal-updates,focal-security,now 1.25.8-2ubuntu0.1 all [installed,automatic]
python3.8-dev/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed]
python3.8-minimal/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed,automatic]
python3.8-venv/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed]
python3.8/focal-updates,focal-security,now 3.8.10-0ubuntu1~20.04.2 amd64 [installed]
python3/focal,now 3.8.2-0ubuntu2 amd64 [installed,automatic]
And setuptools:
Copy code
$ /usr/bin/python3 -c "import setuptools; print(setuptools.__version__)"
60.5.0
Let me know if I’ve missed anything.
Oh you see, I don’t install in the Dockerfile at all
Copy code
python3-setuptools
whereas you do 
apt update && apt install -y python3.8-distutils python3.8-venv python3-setuptools
e
Its the get-pip curl that does it.
👀 1
Thanks.
Ok, I don't repro, but you also said this only happens some of the time - somehow. Is that right?
f
exactly, that’s the weirdest part. I’ve only seen it two times over the thousands of builds we’ve had so far. The first one happened a couple of weeks ago, and today I’ve got myself the second time.
https://bootstrap.pypa.io/get-pip.py seems to be getting the latest 
setuptools
. But maybe due to some other process/tool, a different version of 
setuptools
may be installed each time a Docker image is built?
I’ll run the build/run/print version of setuptools loop tomorrow over a few hours to check the hypothesis of mine
e
Not sure. Either way, Pex tries very hard to guard itself from all system packages. It attempts to scrub them all from the sys.path / sys.modules, etc.
I played with 
SETUPTOOLS_USE_DISTUTILS
and used 
local
and 
stdlib
and neither triggered the issue. Thats a magic env var that has to do with the setuptools 
_distutils_hack
machinery.
Ok, thanks @fresh-cat-90827 more data like that will be useful / critical here.
f
great, give me a few days please, I’ll post later once I have experimented enough. Forever grateful for all the support 🙂
❤️ 1
hey @enough-analyst-54434! So I couldn’t reproduce it locally — it must have been a good chance that in the past the exact same 
apt-get install
command in the 
Dockerfile
resulted in 
python3-setuptools
being installed, but then later it was not the case. Our CI would try to build it (the Docker image) on the same agent it built before, and if the 
Dockerfile
has not changed, it will be cached and thus broken until it’s fixed. So I think we were just playing Docker cache lottery. I’ve added 
python3-setuptools \
to this list of Python related packages and the problem has gone away since then.
Copy code
python3.8-dev \
       python3.8-distutils \
       python3.8-venv \
3 Views
Open in Slack
PreviousNext
Powered by