sparse-lifeguard-95737
10/31/2022, 7:09 PMModuleNotFoundError: No module named 'distutils.util'
in some CI steps 🧵Traceback (most recent call last):
File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/pex", line 243, in <module>
runpy.run_module(module_name, run_name="__main__", alter_sys=True)
File "/usr/lib/python3.8/runpy.py", line 207, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/__pex_patched_pip__.py", line 10, in <module>
runpy.run_module(mod_name="pip", run_name="__main__", alter_sys=True)
File "/usr/lib/python3.8/runpy.py", line 207, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/__main__.py", line 23, in <module>
from pip._internal.cli.main import main as _main # isort:skip # noqa
File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 10, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
from pip._internal.cli import cmdoptions
File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 18, in <module>
from distutils.util import strtobool
ModuleNotFoundError: No module named 'distutils.util'
pex
?enough-analyst-54434
10/31/2022, 7:12 PMsparse-lifeguard-95737
10/31/2022, 7:13 PMRUN curl -Lo /tmp/pyenv.tar.gz <https://github.com/pyenv/pyenv/archive/refs/tags/v2.3.3.tar.gz> \
&& echo '2a6093c922d2a420b5ae45143ab973b0a85ee486c408cb487188b64edadfab35 /tmp/pyenv.tar.gz' | sha256sum -c \
&& mkdir -p /tmp/pyenv && tar xzf /tmp/pyenv.tar.gz -C /tmp/pyenv --strip-components 1 && rm /tmp/pyenv.tar.gz \
&& /tmp/pyenv/plugins/python-build/install.sh \
&& rm -r /tmp/pyenv
# `MAKE_OPTS` and `PYTHON_CONFIGURE_OPTS` are lifted from the official `python` image build.
# <https://github.com/docker-library/python/blob/master/3.8/slim-buster/Dockerfile#L69-L81>
#
# NOTE: We install to /opt/python/<version> instead of the default /usr/local to make copying
# the outputs into our final build stage very easy.
RUN mkdir -p "/opt/python/${PYTHON_VERSION}" \
arch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
&& export arch \
&& MAKE_OPTS="-j $(nproc) LDFLAGS='-Wl,--strip-all'" \
PYTHON_CONFIGURE_OPTS="--build=${arch} --enable-loadable-sqlite-extensions --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip ${EXTRA_PYTHON_BUILD_FLAGS}" \
python-build --verbose "${PYTHON_VERSION}" "/opt/python/${PYTHON_VERSION}"
enough-analyst-54434
10/31/2022, 7:15 PM/opt/python/${PYTHON_VERSION}/lib/python3.8/distutils/
sparse-lifeguard-95737
10/31/2022, 7:19 PMroot@01e739af8e89:/# ls /opt/python/3.8.13/lib/python3.8/distutils/
archive_util.py config.py dir_util.py filelist.py _msvccompiler.py text_file.py
bcppcompiler.py core.py dist.py file_util.py msvccompiler.py unixccompiler.py
ccompiler.py cygwinccompiler.py errors.py __init__.py README util.py
cmd.py debug.py extension.py log.py spawn.py versionpredicate.py
command dep_util.py fancy_getopt.py msvc9compiler.py sysconfig.py version.py
enough-analyst-54434
10/31/2022, 7:23 PM/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/distutils
venv? Did you confirm its not there? What is there?sparse-lifeguard-95737
10/31/2022, 7:24 PMenough-analyst-54434
10/31/2022, 7:26 PMSETUPTOOLS_USE_DISTUTILS=stdlib
may be a thing to try. That is just mashing on the keyboard though. If that does help then comes the work of understanding why.sparse-lifeguard-95737
10/31/2022, 7:28 PMsetuptools==45.3.0
in our 3rdparty/requirements.txt
- is it possible that by pinning the older versions we’re messing up some expectation?./pants help setuptools
still shows current value: setuptools>=63.1.0,<64.0
but I’m not sure how the two might bleed togetherenough-analyst-54434
10/31/2022, 7:29 PMsparse-lifeguard-95737
10/31/2022, 7:41 PMpublic.ecr.aws/docker/library/ubuntu
(mirror of ubuntu
from DockerHub)which python
shows nothing in the base imageenough-analyst-54434
10/31/2022, 7:45 PM_PEX_FILE_LOCK_STYLE=bsd
I added for your other missing files case assuming you bump Pex to 2.1.112. I think I noted the details on a ticket.sparse-lifeguard-95737
10/31/2022, 7:53 PM[subprocess-environment]
env_vars.add = ["SETUPTOOLS_USE_DISTUTILS=stdlib"]
to pants.toml
didn’t fix itsleep
into the step that’s been consistently hitting this on my upgrade PR - will be able to jump in and look at the venv pretty easilyenough-analyst-54434
10/31/2022, 9:38 PMSETUPTOOLS_USE_DISTUTILS=stdlib
- I'd think the `SETUPTOOLS_USE_DISTUTILS=local`would be the one to work if any. You have indirect proof the stdlib version of the file does not exist. And local
here refers to setuptools vendored distutils.sparse-lifeguard-95737
10/31/2022, 9:38 PMlib/python3.8
is site-packages
runner@clr-runner-pwqwt-6kpk8:~/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6$ ls lib/python3.8/
site-packages
local
nowenough-analyst-54434
10/31/2022, 9:56 PMsparse-lifeguard-95737
11/01/2022, 1:03 AMlocal
eitherenough-analyst-54434
11/01/2022, 1:21 AMsparse-lifeguard-95737
11/01/2022, 12:34 PMIgnoring the following environment variables in Pex venv mode:
_PEX_FILE_LOCK_STYLE=bsd
N
requirement(s) for `<pex>`” step - the actual CLI commands look like:
/home/runner/.pants/named_caches/pex_root/venvs/ab8aed8139d5620758699080988da0da5d11ec29/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/runner/.pants/named_caches/pex_root/venvs/ab8aed8139d5620758699080988da0da5d11ec29/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/runner/.pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /home/runner/.pants/named_caches/pex_root/built_wheels/sdists/peewee-3.15.2.tar.gz/94d8eb7fc2b8d97d6216382bf5961239645dc9e2473b0e732eb6c79c920c4768/cp38-cp38-manylinux_2_31_x86_64.workdir /home/runner/.pants/named_caches/pex_root/downloads/94d8eb7fc2b8d97d6216382bf5961239645dc9e2473b0e732eb6c79c920c4768/peewee-3.15.2.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15
Engine traceback:
in select
..
in pants.core.goals.run.run
`run` goal
in pants.backend.python.goals.run_pex_binary.create_pex_binary_run_request
..
in pants.backend.python.goals.package_pex_binary.package_pex_binary
..
in pants.backend.python.util_rules.pex.create_pex
..
in pants.backend.python.util_rules.pex.build_pex
..
in pants.engine.process.fallible_to_exec_result_or_raise
..
setuptools
version, but also a tiny one that only contains semgrep==0.111.1
package
PEXes for inclusion in docker images. it hits the tiny resolve when we try to ./pants run
a pex_binary
target.
We have `3rdparty/semgrep/BUILD.pants`:
python_requirement(
name="lib",
requirements=["semgrep==0.111.1"],
resolve="semgrep",
)
pex_binary(
name="bin",
dependencies=[":lib"],
resolve="semgrep",
entry_point="semgrep.__main__:main", # lifted from <https://github.com/returntocorp/semgrep/blob/develop/cli/setup.py#L148>
)
and then we try to ./pants run 3rdparty/semgrep:bin
, and hit the errorenough-analyst-54434
11/01/2022, 3:33 PMFROM public.ecr.aws/docker/library/ubuntu
RUN curl -Lo /tmp/pyenv.tar.gz <https://github.com/pyenv/pyenv/archive/refs/tags/v2.3.3.tar.gz> \
&& echo '2a6093c922d2a420b5ae45143ab973b0a85ee486c408cb487188b64edadfab35 /tmp/pyenv.tar.gz' | sha256sum -c \
&& mkdir -p /tmp/pyenv && tar xzf /tmp/pyenv.tar.gz -C /tmp/pyenv --strip-components 1 && rm /tmp/pyenv.tar.gz \
&& /tmp/pyenv/plugins/python-build/install.sh \
&& rm -r /tmp/pyenv
# `MAKE_OPTS` and `PYTHON_CONFIGURE_OPTS` are lifted from the official `python` image build.
# <https://github.com/docker-library/python/blob/master/3.8/slim-buster/Dockerfile#L69-L81>
#
# NOTE: We install to /opt/python/<version> instead of the default /usr/local to make copying
# the outputs into our final build stage very easy.
RUN mkdir -p "/opt/python/${PYTHON_VERSION}" \
arch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
&& export arch \
&& MAKE_OPTS="-j $(nproc) LDFLAGS='-Wl,--strip-all'" \
PYTHON_CONFIGURE_OPTS="--build=${arch} --enable-loadable-sqlite-extensions --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip ${EXTRA_PYTHON_BUILD_FLAGS}" \
python-build --verbose "${PYTHON_VERSION}" "/opt/python/${PYTHON_VERSION}"
Do you use just a bare public.ecr.aws/docker/library/ubuntu
or is there a tag?sparse-lifeguard-95737
11/01/2022, 3:33 PMpublic.ecr.aws/docker/library/ubuntu:20.04
ubuntu:20.04
enough-analyst-54434
11/01/2022, 3:44 PMsparse-lifeguard-95737
11/01/2022, 4:40 PMDockerfile
for our base-python
image:
FROM ubuntu:20.04 AS ubuntu
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y --no-install-recommends locales && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
locale-gen en_US.UTF-8 && \
update-locale en_US.UTF-8
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
FROM ubuntu AS python-build
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN apt-get update && \
apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
dpkg-dev \
libbz2-dev \
libffi-dev \
liblzma-dev \
libncursesw5-dev \
libreadline-dev \
libsqlite3-dev \
libssl-dev \
libxml2-dev \
libxmlsec1-dev \
llvm \
make \
tk-dev \
wget \
xz-utils \
zlib1g-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
RUN curl -Lo /tmp/pyenv.tar.gz <https://github.com/pyenv/pyenv/archive/refs/tags/v2.3.3.tar.gz> \
&& echo '2a6093c922d2a420b5ae45143ab973b0a85ee486c408cb487188b64edadfab35 /tmp/pyenv.tar.gz' | sha256sum -c \
&& mkdir -p /tmp/pyenv && tar xzf /tmp/pyenv.tar.gz -C /tmp/pyenv --strip-components 1 && rm /tmp/pyenv.tar.gz \
&& /tmp/pyenv/plugins/python-build/install.sh \
&& rm -r /tmp/pyenv
RUN mkdir -p /opt/python/3.8.13 \
arch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
&& export arch \
&& MAKE_OPTS="-j $(nproc) LDFLAGS='-Wl,--strip-all'" \
PYTHON_CONFIGURE_OPTS="--build=${arch} --enable-loadable-sqlite-extensions --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip --enable-optimizations" \
python-build --verbose 3.8.13 /opt/python/3.8.13
RUN find /opt/python/3.8.13 -depth -type f -name '*.a' -exec rm '{}' \;
RUN find /opt/python/3.8.13 -depth -type d -name __pycache__ -exec rm -r '{}' \;
RUN find /opt/python/3.8.13 -depth -type d -a \( -name test -o -name tests -o -name idle_test \) -exec rm -r '{}' \;
RUN find /opt/python/3.8.13 -depth -type f -name '*.exe' -exec rm '{}' \;
FROM ubuntu AS final
COPY --from=python-build /opt/python/3.8.13 /opt/python/3.8.13
RUN for prog in /opt/python/3.8.13/bin/*; do ln -s "$prog" "/usr/local/bin/$(basename "$prog")"; done
RUN apt-get update && \
apt-get install -y --no-install-recommends libexpat1 && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
but! the error doesn’t repro locally in itenough-analyst-54434
11/01/2022, 4:43 PMsparse-lifeguard-95737
11/01/2022, 4:56 PMpython3*
packages in our final CI imageenough-analyst-54434
11/01/2022, 4:58 PMsparse-lifeguard-95737
11/01/2022, 4:58 PMapt install python3-distutils
fixes it for me locallyenough-analyst-54434
11/01/2022, 4:59 PMsparse-lifeguard-95737
11/01/2022, 4:59 PMenough-analyst-54434
11/01/2022, 4:59 PMsparse-lifeguard-95737
11/01/2022, 5:00 PM/usr/local/bin/python -> /opt/python/<version>/bin/python
(same with other binaries from the install)
2. set PANTS_PYTHON_BOOTSTRAP_SEARCH_PATH="['<PATH>']"
enough-analyst-54434
11/01/2022, 5:00 PMsparse-lifeguard-95737
11/01/2022, 5:01 PMwhich -a python
only shows the symlink to /opt
FWIWwhich -a python3
shows moreenough-analyst-54434
11/01/2022, 5:02 PMsparse-lifeguard-95737
11/01/2022, 5:02 PMenough-analyst-54434
11/01/2022, 5:04 PMsparse-lifeguard-95737
11/01/2022, 5:04 PMthis is sorted / sortable by you?
enough-analyst-54434
11/01/2022, 5:04 PMsparse-lifeguard-95737
11/01/2022, 5:05 PM/opt
is already first in the PATH
enough-analyst-54434
11/01/2022, 5:05 PMsparse-lifeguard-95737
11/01/2022, 5:06 PMenough-analyst-54434
11/01/2022, 5:08 PMsparse-lifeguard-95737
11/01/2022, 5:44 PM