https://pantsbuild.org/ logo
Powered by
s

sparse-lifeguard-95737

10/31/2022, 7:09 PM
on 2.15.0a0, I’m seeing 
ModuleNotFoundError: No module named 'distutils.util'
in some CI steps 🧵
example stack trace:
Copy code
Traceback (most recent call last):
  File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/pex", line 243, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/__pex_patched_pip__.py", line 10, in <module>
    runpy.run_module(mod_name="pip", run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/__main__.py", line 23, in <module>
    from pip._internal.cli.main import main as _main  # isort:skip # noqa
  File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 18, in <module>
    from distutils.util import strtobool
ModuleNotFoundError: No module named 'distutils.util'
this is using the same runtime environment as before Pants 2.15, so I don’t think it’s an issue of a broken python install
@enough-analyst-54434 could this be a bug in newer 
pex
?
e

enough-analyst-54434

10/31/2022, 7:12 PM
Perhaps. What is the underlying image? Is the Python 3.8 stock from that image's package repo?
s

sparse-lifeguard-95737

10/31/2022, 7:13 PM
no, it’s a private image that builds python from source to avoid the strangeness of system package repo python
not the full Dockerfile, but the relevant lines for building Python:
Copy code
RUN curl -Lo /tmp/pyenv.tar.gz <https://github.com/pyenv/pyenv/archive/refs/tags/v2.3.3.tar.gz> \
    && echo '2a6093c922d2a420b5ae45143ab973b0a85ee486c408cb487188b64edadfab35  /tmp/pyenv.tar.gz' | sha256sum -c \
    && mkdir -p /tmp/pyenv && tar xzf /tmp/pyenv.tar.gz -C /tmp/pyenv --strip-components 1 && rm /tmp/pyenv.tar.gz \
    && /tmp/pyenv/plugins/python-build/install.sh \
    && rm -r /tmp/pyenv

# `MAKE_OPTS` and `PYTHON_CONFIGURE_OPTS` are lifted from the official `python` image build.
# <https://github.com/docker-library/python/blob/master/3.8/slim-buster/Dockerfile#L69-L81>
#
# NOTE: We install to /opt/python/<version> instead of the default /usr/local to make copying
# the outputs into our final build stage very easy.
RUN mkdir -p "/opt/python/${PYTHON_VERSION}" \
    arch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
    && export arch \
    && MAKE_OPTS="-j $(nproc) LDFLAGS='-Wl,--strip-all'" \
    PYTHON_CONFIGURE_OPTS="--build=${arch} --enable-loadable-sqlite-extensions --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip ${EXTRA_PYTHON_BUILD_FLAGS}" \
    python-build --verbose "${PYTHON_VERSION}" "/opt/python/${PYTHON_VERSION}"
e

enough-analyst-54434

10/31/2022, 7:15 PM
Ok. So what's in /home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/distutils/ ?
👀 1
So, there's this ~well-known issue with Debian python that you need to install pythonX-distutils explicitly.
They do not include it in their standard Python packages.
Better would be what is in the source prefix 
/opt/python/${PYTHON_VERSION}/lib/python3.8/distutils/
👀 1
s

sparse-lifeguard-95737

10/31/2022, 7:19 PM
Copy code
root@01e739af8e89:/# ls /opt/python/3.8.13/lib/python3.8/distutils/
archive_util.py  config.py           dir_util.py      filelist.py       _msvccompiler.py  text_file.py
bcppcompiler.py  core.py             dist.py          file_util.py      msvccompiler.py   unixccompiler.py
ccompiler.py     cygwinccompiler.py  errors.py        __init__.py       README            util.py
cmd.py           debug.py            extension.py     log.py            spawn.py          versionpredicate.py
command          dep_util.py         fancy_getopt.py  msvc9compiler.py  sysconfig.py      version.py
it seems to consistently repro in our Linux CI env, but doesn’t repro on my Mac
e

enough-analyst-54434

10/31/2022, 7:23 PM
Ok, so util.py is in the base CPython distribution. What about the 
/home/runner/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6/lib/python3.8/distutils
venv? Did you confirm its not there? What is there?
s

sparse-lifeguard-95737

10/31/2022, 7:24 PM
that one will unfortunately take more time to check, since our CI runners are ephemeral - since it didn’t repro on my laptop I’ll need to grab a runner mid-run to poke around (sigh)
will let you know what I find
e

enough-analyst-54434

10/31/2022, 7:26 PM
Urg, yeah - right. Thank you. The only relevant facts I can think of are that Pex got even more hermetic in recent releases and that distutils and setuptools have been doing a dance: https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html So, playing with exporting 
SETUPTOOLS_USE_DISTUTILS=stdlib
may be a thing to try. That is just mashing on the keyboard though. If that does help then comes the work of understanding why.
s

sparse-lifeguard-95737

10/31/2022, 7:28 PM
we do have 
setuptools==45.3.0
in our 
3rdparty/requirements.txt
- is it possible that by pinning the older versions we’re messing up some expectation?
./pants help setuptools
still shows 
current value: setuptools>=63.1.0,<64.0
but I’m not sure how the two might bleed together
e

enough-analyst-54434

10/31/2022, 7:29 PM
No clue. Pex's own vendored Pip / setuptools are at 44.0.0
👍 1
I think thrashing around on version numbers won't be super helpful yet. The key is to look at backtrace and figure out which version is relevant in that backtrace ... once we get there. 1st trying `SETUPTOOLS_USE_DISTUTILS={local,stdlib}`since its easy would be good.
Just for a sanity check - so your image is either not Debian-based, or if it is you have 0 Pythons installed from that image base - only your own in /opt ?
s

sparse-lifeguard-95737

10/31/2022, 7:41 PM
it’s based off of 
public.ecr.aws/docker/library/ubuntu
(mirror of 
ubuntu
from DockerHub)
checking what comes pre-installed on that one now
yeah, 
which python
shows nothing in the base image
e

enough-analyst-54434

10/31/2022, 7:45 PM
Ok. Then the data from the named_caches venv and / or SETUPTOOLS_USE_DISTUTILS try would be useful. Or ... trying out the 
_PEX_FILE_LOCK_STYLE=bsd
I added for your other missing files case assuming you bump Pex to 2.1.112. I think I noted the details on a ticket.
s

sparse-lifeguard-95737

10/31/2022, 7:53 PM
yeah I was drilling into the pytest-batching work so haven’t had the chance to try the new pex yet, will check the ticket now
adding:
Copy code
[subprocess-environment]
env_vars.add = ["SETUPTOOLS_USE_DISTUTILS=stdlib"]
to 
pants.toml
didn’t fix it
threw a 
sleep
into the step that’s been consistently hitting this on my upgrade PR - will be able to jump in and look at the venv pretty easily
e

enough-analyst-54434

10/31/2022, 9:38 PM
Great - thank you. On the 
SETUPTOOLS_USE_DISTUTILS=stdlib
- I'd think the `SETUPTOOLS_USE_DISTUTILS=local`would be the one to work if any. You have indirect proof the stdlib version of the file does not exist. And 
local
here refers to setuptools vendored distutils.
s

sparse-lifeguard-95737

10/31/2022, 9:38 PM
ah, ok - will give that one a try too
Ok so when the error hits, there is no distutils in the venv. the only thing under 
lib/python3.8
is 
site-packages
Copy code
runner@clr-runner-pwqwt-6kpk8:~/.pants/named_caches/pex_root/venvs/29638d5feb42d02ca1bfbcc777675dd78fc662d7/ddab8011daaee380698ac2fb9701af18c90c03f6$ ls lib/python3.8/
site-packages
trying with 
local
now
e

enough-analyst-54434

10/31/2022, 9:56 PM
Ok, that's interesting data - so the whole non-frozen stdlib is missing then.
s

sparse-lifeguard-95737

11/01/2022, 1:03 AM
no luck with 
local
either
e

enough-analyst-54434

11/01/2022, 1:21 AM
Ok, thanks for mashing on the keyboard, I did not expect that might help. After you try the bsd locks I'd like to regroup if needed and dig more.
s

sparse-lifeguard-95737

11/01/2022, 12:34 PM
new pex + bsd locks also didn’t help
oh, although I see now in the logs:
Copy code
Ignoring the following environment variables in Pex venv mode:
_PEX_FILE_LOCK_STYLE=bsd
the failures are hitting in the “Building 
N
requirement(s) for `<pex>`” step - the actual CLI commands look like:
Copy code
/home/runner/.pants/named_caches/pex_root/venvs/ab8aed8139d5620758699080988da0da5d11ec29/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/runner/.pants/named_caches/pex_root/venvs/ab8aed8139d5620758699080988da0da5d11ec29/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/runner/.pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /home/runner/.pants/named_caches/pex_root/built_wheels/sdists/peewee-3.15.2.tar.gz/94d8eb7fc2b8d97d6216382bf5961239645dc9e2473b0e732eb6c79c920c4768/cp38-cp38-manylinux_2_31_x86_64.workdir /home/runner/.pants/named_caches/pex_root/downloads/94d8eb7fc2b8d97d6216382bf5961239645dc9e2473b0e732eb6c79c920c4768/peewee-3.15.2.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15
and engine traceback:
Copy code
Engine traceback:
  in select
    ..
  in pants.core.goals.run.run
    `run` goal
  in pants.backend.python.goals.run_pex_binary.create_pex_binary_run_request
    ..
  in pants.backend.python.goals.package_pex_binary.package_pex_binary
    ..
  in pants.backend.python.util_rules.pex.create_pex
    ..
  in pants.backend.python.util_rules.pex.build_pex
    ..
  in pants.engine.process.fallible_to_exec_result_or_raise
    ..
I missed this yesterday, but we’re actually seeing the error across multiple resolves. our “main” one that pins a 
setuptools
version, but also a tiny one that only contains 
semgrep==0.111.1
the error hits our main resolve when trying to 
package
PEXes for inclusion in docker images. it hits the tiny resolve when we try to 
./pants run
a 
pex_binary
target. We have `3rdparty/semgrep/BUILD.pants`:
Copy code
python_requirement(
    name="lib",
    requirements=["semgrep==0.111.1"],
    resolve="semgrep",
)

pex_binary(
    name="bin",
    dependencies=[":lib"],
    resolve="semgrep",
    entry_point="semgrep.__main__:main",  # lifted from <https://github.com/returntocorp/semgrep/blob/develop/cli/setup.py#L148>
)
and then we try to 
./pants run 3rdparty/semgrep:bin
, and hit the error
oh - I’m able to reproduce locally! not on my mac, but running in the linux container on my mac
e

enough-analyst-54434

11/01/2022, 3:33 PM
Ok, great. Thanks @sparse-lifeguard-95737. It sounds like I should be able to repro to with:
Copy code
FROM public.ecr.aws/docker/library/ubuntu

RUN curl -Lo /tmp/pyenv.tar.gz <https://github.com/pyenv/pyenv/archive/refs/tags/v2.3.3.tar.gz> \
    && echo '2a6093c922d2a420b5ae45143ab973b0a85ee486c408cb487188b64edadfab35  /tmp/pyenv.tar.gz' | sha256sum -c \
    && mkdir -p /tmp/pyenv && tar xzf /tmp/pyenv.tar.gz -C /tmp/pyenv --strip-components 1 && rm /tmp/pyenv.tar.gz \
    && /tmp/pyenv/plugins/python-build/install.sh \
    && rm -r /tmp/pyenv

# `MAKE_OPTS` and `PYTHON_CONFIGURE_OPTS` are lifted from the official `python` image build.
# <https://github.com/docker-library/python/blob/master/3.8/slim-buster/Dockerfile#L69-L81>
#
# NOTE: We install to /opt/python/<version> instead of the default /usr/local to make copying
# the outputs into our final build stage very easy.
RUN mkdir -p "/opt/python/${PYTHON_VERSION}" \
    arch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
    && export arch \
    && MAKE_OPTS="-j $(nproc) LDFLAGS='-Wl,--strip-all'" \
    PYTHON_CONFIGURE_OPTS="--build=${arch} --enable-loadable-sqlite-extensions --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip ${EXTRA_PYTHON_BUILD_FLAGS}" \
    python-build --verbose "${PYTHON_VERSION}" "/opt/python/${PYTHON_VERSION}"
Do you use just a bare 
public.ecr.aws/docker/library/ubuntu
or is there a tag?
s

sparse-lifeguard-95737

11/01/2022, 3:33 PM
public.ecr.aws/docker/library/ubuntu:20.04
if you aren’t set up with the docker ECR helper you might get auth errors (even though it’s public, sigh). it should be identical to 
ubuntu:20.04
I omitted some system deps, let me paste the whole dockerfile (nothing secret)
e

enough-analyst-54434

11/01/2022, 3:44 PM
Ok, great. I won't have time until this evening to flip to this, but, afaict, I'll have what I need.
s

sparse-lifeguard-95737

11/01/2022, 4:40 PM
errrrrr, so this is the cleaned up 
Dockerfile
for our 
base-python
image:
Copy code
FROM ubuntu:20.04 AS ubuntu
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y --no-install-recommends locales && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    locale-gen en_US.UTF-8 && \
    update-locale en_US.UTF-8

ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8

FROM ubuntu AS python-build
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        build-essential \
        ca-certificates \
        curl \
        dpkg-dev \
        libbz2-dev \
        libffi-dev \
        liblzma-dev \
        libncursesw5-dev \
        libreadline-dev \
        libsqlite3-dev \
        libssl-dev \
        libxml2-dev \
        libxmlsec1-dev \
        llvm \
        make \
        tk-dev \
        wget \
        xz-utils \
        zlib1g-dev && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN curl -Lo /tmp/pyenv.tar.gz <https://github.com/pyenv/pyenv/archive/refs/tags/v2.3.3.tar.gz> \
    && echo '2a6093c922d2a420b5ae45143ab973b0a85ee486c408cb487188b64edadfab35  /tmp/pyenv.tar.gz' | sha256sum -c \
    && mkdir -p /tmp/pyenv && tar xzf /tmp/pyenv.tar.gz -C /tmp/pyenv --strip-components 1 && rm /tmp/pyenv.tar.gz \
    && /tmp/pyenv/plugins/python-build/install.sh \
    && rm -r /tmp/pyenv

RUN mkdir -p /opt/python/3.8.13 \
    arch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
    && export arch \
    && MAKE_OPTS="-j $(nproc) LDFLAGS='-Wl,--strip-all'" \
    PYTHON_CONFIGURE_OPTS="--build=${arch} --enable-loadable-sqlite-extensions --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip --enable-optimizations" \
    python-build --verbose 3.8.13 /opt/python/3.8.13

RUN find /opt/python/3.8.13 -depth -type f -name '*.a' -exec rm '{}' \;
RUN find /opt/python/3.8.13 -depth -type d -name __pycache__ -exec rm -r '{}' \;
RUN find /opt/python/3.8.13 -depth -type d -a \( -name test -o -name tests -o -name idle_test \) -exec rm -r '{}' \;
RUN find /opt/python/3.8.13 -depth -type f -name '*.exe' -exec rm '{}' \;

FROM ubuntu AS final

COPY --from=python-build /opt/python/3.8.13 /opt/python/3.8.13
RUN for prog in /opt/python/3.8.13/bin/*; do ln -s "$prog" "/usr/local/bin/$(basename "$prog")"; done

RUN apt-get update && \
    apt-get install -y --no-install-recommends libexpat1 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
but! the error doesn’t repro locally in it
which suggests one of the packages we install when we build on our base-python image to produce our actions-runner image is the culprit
oh no wait I wasn’t running on the 2.15 branch
🤦‍♂️
checking it the right way this time…
e

enough-analyst-54434

11/01/2022, 4:43 PM
Ok. Thanks for digging. I'll hang back then until I hear more.
👍 1
s

sparse-lifeguard-95737

11/01/2022, 4:56 PM
yeahhh something seems to be installing the 
python3*
packages in our final CI image
but - we didn’t make any changes to the CI image between Pants 2.14 and 2.15, so if those packages are the problem then something changed in Pants/pex
e

enough-analyst-54434

11/01/2022, 4:58 PM
Pex got more strict
s

sparse-lifeguard-95737

11/01/2022, 4:58 PM
hmmm
confirmed that 
apt install python3-distutils
fixes it for me locally
e

enough-analyst-54434

11/01/2022, 4:59 PM
Ok, so that says Pex is using the "wrong" interpreter now.
s

sparse-lifeguard-95737

11/01/2022, 4:59 PM
yea
e

enough-analyst-54434

11/01/2022, 4:59 PM
How do you arrange for the right interpreter to be used? The one in /opt?
s

sparse-lifeguard-95737

11/01/2022, 5:00 PM
we: 1. symlink 
/usr/local/bin/python -> /opt/python/<version>/bin/python
(same with other binaries from the install) 2. set 
PANTS_PYTHON_BOOTSTRAP_SEARCH_PATH="['<PATH>']"
e

enough-analyst-54434

11/01/2022, 5:00 PM
The bootstrap is just the bootstrap. Pex will later find anything on PATH
Let me see if there was a Pants change to bootstrap. That's useful new info from you.
s

sparse-lifeguard-95737

11/01/2022, 5:01 PM
which -a python
only shows the symlink to 
/opt
FWIW
ohhhh but 
which -a python3
shows more
e

enough-analyst-54434

11/01/2022, 5:02 PM
Yes
SO its really not clear to me why this worked before - but it's always been a time bomb.
s

sparse-lifeguard-95737

11/01/2022, 5:02 PM
luck of the draw
e

enough-analyst-54434

11/01/2022, 5:04 PM
So, I think this is sorted / sortable by you? If so, at some point I'd like to rewind the stack to the original Pex issue / bsd attempt and see if missing PEX modules are fixed by that. This case being confusingly different and just plain old missing distro files for the base interpreter.
s

sparse-lifeguard-95737

11/01/2022, 5:04 PM
what do you mean by “this” in:
Copy code
this is sorted / sortable by you?
e

enough-analyst-54434

11/01/2022, 5:04 PM
Afaict there was only a change to docs about the python bootstrap in https://github.com/pantsbuild/pants/pull/17063
Well, you can ensure the image doesn't have bad python installed in it if you don't want it.
👍 1
s

sparse-lifeguard-95737

11/01/2022, 5:05 PM
the install from 
/opt
is already first in the 
PATH
e

enough-analyst-54434

11/01/2022, 5:05 PM
Pex picks latest patch IIRC
What is the /opt 3.8.x vs system 3.8.y ?
s

sparse-lifeguard-95737

11/01/2022, 5:06 PM
3.8.13 vs 3.8.10
e

enough-analyst-54434

11/01/2022, 5:08 PM
Ok, and I checked the code and my recollection is correct. So for an interpreter constraint with wiggle room, 3.8.13 should be picked over 3.8.10.
Let me hunt around a bit ...
s

sparse-lifeguard-95737

11/01/2022, 5:44 PM
in the meantime I’ve found how the system python is sneaking in and it looks like an easy fix to rip out
Powered by