https://pantsbuild.org/ logo
#general
Title
# general
p

polite-angle-19513

03/30/2023, 3:13 PM
Greetings! I'm attempting to test multiple interpreters. Using pants 2.15.0. Want to support python 3.8, 3.9 and 3.10 in testing. Ubuntu 22.04 with 3.10 native. Used Deadsnake to install python3.8 and python3.9. pants.toml
Copy code
[python]
  interpreter_constraints = [">=3.8,  <3.11"]
From project tests BUILD
Copy code
python_tests(
    name="tests",
    dependencies=[
        "src/python/project-lib:lib",
    ],
    sources=[
        "**/*_test.py",
    ],
    interpreter_constraints=parametrize(py38=["==3.8.*"], py39=["==3.9.*"], py310=["==3.10.*"]),
)
My general flow:
Copy code
pants --keep-sandboxes=on_failure lint src/python/project-lib:lib
pants --keep-sandboxes=on_failure check src/python/project-lib:lib
pants --keep-sandboxes=on_failure test test/python/project-lib:tests -- -s
The lint and check goals during execution I'm getting errors when pants attempts to build the requirements.pex and the pip.log doesn't exist (only showing error from *lint*; the check goal it's during mypy.pex building):
Copy code
11:05:27.77 [INFO] Canceled: Building pylint.pex from pylint_default.lock
11:05:28.38 [INFO] Preserving local process execution dir /tmp/pants-sandbox-3ffJgp for Building requirements.pex with 7 requirements: backoff~=2.2.1; python_version >= "3.7" and python_version < "4.0", jsonpickle~=3.0.1; python_version >= "3.7", loguru~=0.6.0; python_version >= "3.5", pandas~=1.5.3; python_version >= "3.8", pendulum~=2.1.2; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", python-magic~=0.4.27; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", requests~=2.28.2; python_version >= "3.7" and python_version < "4"
11:05:28.38 [INFO] Completed: Building requirements.pex with 7 requirements: backoff~=2.2.1; python_version >= "3.7" and python_version < "4.0", jsonpickle~=3.0.1; python_version >= "3.7", loguru~=0.6.0; python_version >= "3.5", p... (334 characters truncated)
11:05:28.39 [ERROR] 1 Exception encountered:

Engine traceback:
  in `lint` goal
  in Lint using Pylint

ProcessExecutionFailure: Process 'Building requirements.pex with 7 requirements: backoff~=2.2.1; python_version >= "3.7" and python_version < "4.0", jsonpickle~=3.0.1; python_version >= "3.7", loguru~=0.6.0; python_version >= "3.5", pandas~=1.5.3; python_version >= "3.8", pendulum~=2.1.2; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", python-magic~=0.4.27; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0", requests~=2.28.2; python_version >= "3.7" and python_version < "4"' failed with exit code 1.
stdout:

stderr:
[Errno 2] No such file or directory: '/tmp/pants-sandbox-3ffJgp/.tmp/pex-pip-log.2s_u5pjx/pip.log'
The test goal is a bit different, failing on building *pytest.pe*x with a downloading issue:
Copy code
11:06:21.00 [INFO] Canceled: Building pytest.pex from pytest_default.lock
11:06:21.67 [INFO] Preserving local process execution dir /tmp/pants-sandbox-TEAYuo for Building pytest.pex from pytest_default.lock
11:06:21.67 [INFO] Completed: Building pytest.pex from pytest_default.lock
11:06:21.67 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal
  in Run Pytest - test/python/hank-ai-lib/util_util_test.py:tests@interpreter_constraints=py39

ProcessExecutionFailure: Process 'Building pytest.pex from pytest_default.lock' failed with exit code 1.
stdout:

stderr:
There was 1 error downloading required artifacts:
1. coverage 6.4.3 from <https://files.pythonhosted.org/packages/a5/6a/5a201c1b85c5c406c5ee5be4d17c223ad71a5f77937fe9a680b02e6a1fb3/coverage-6.4.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl>
    ModuleNotFoundError: No module named 'distutils.util'
First time attempting to test multiple interpreters. Pants is supposed to pull in all the deps in the sandbox for the particular interpreter version correct? Am I grossly missing something to support multiple interpreter versions for testing? Any and all help is greatly appreciated. Thanks.
e

enough-analyst-54434

03/30/2023, 3:18 PM
The second error is due to the "well known" packaging of Python by Debian. It comes in multiple packages. IIRC you need
python-distutils
and maybe
python-venv
and
python-dev
. That's just for the default Python version on the distro, best to use
pythonX.Y-...
to be explicit though.
How about try that 1st and then report back. The 1st error is extremely strange.
Here is Pex's attempt to document the general form of the second issue fwiw: https://github.com/pantsbuild/pex/issues/1027#issuecomment-684166485
p

polite-angle-19513

03/30/2023, 3:28 PM
Thanks for quick and considered replies. I've installed the python3.(8|9)-distutils and the first and second errors have abated.