lively-gpu-26436
03/01/2023, 7:27 PM["==3.11.*"]
in pants.toml
and a default resolve with some dependencies, all worked perfectly here, the issue comes when adding a py2/py3 library, I changed the interpreter constraints in pants.toml
to ["==2.7.*", ">=3.9"]
and used it as the interpreter constraints for my newly added package too, created a new resolve called py2-py3 = resolves/py2-py3.lock
and re-generated lock files. The problem comes when running the check
goal on my previously working python3 only code, it takes quite a lot of time and finally errors. My question is if someone knows how adding a package with mixed interpreter constraints can cause this check goal to fail in other packages with different constraints that were previously working?
./pants check src/python::
20:19:50.13 [INFO] Completed: Building 4 requirements for requirements.pex from the resolves/python-default.lock resolve: aiohttp[speedups], prometheus-client, pyarrow, pydantic
20:19:50.13 [ERROR] 1 Exception encountered:
Engine traceback:
in `check` goal
in Typecheck using MyPy - mypy
ProcessExecutionFailure: Process 'Building 4 requirements for requirements.pex from the resolves/python-default.lock resolve: aiohttp[speedups], prometheus-client, pyarrow, pydantic' failed with exit code 1.
stdout:
stderr:
There was 1 error downloading required artifacts:
1. pyarrow 0.16 from <https://files.pythonhosted.org/packages/3e/00/4e099e2f1992c555771090b5971c59751fc152c68270a31bfeabd114c068/pyarrow-0.16.0.tar.gz>
ERROR: Command errored out with exit status 1: /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/791e686d7c6a49493bab896fbb2645175e97bb31/bin/python /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/791e686d7c6a49493bab896fbb2645175e97bb31/lib/python3.11/site-packages/pip install --ignore-installed --no-user --prefix /Users/anler/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-build-env-fa06xvk8/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i <https://pypi.org/simple/> -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel Check the logs for full command output.
Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
python_requirements
target accepts an interpreter_constraints key but it doesnātpants.toml
interpreter_constraints be a superset of the ones used by my library? If I try to keep my default interpreter_constraints in pants.toml
to say >=3.9
, then if I try to run the tests of my python2/3 package which is defined as:
python_tests(
name="tests",
resolve="py2-py3",
interpreter_constraints=parametrize(py2=["==2.7.*"], py3=[">=3.9"]),
)
then it errors with:
InvalidLockfileError: You are using the lockfile at resolves/py2-py3.lock to install the resolve `py2-py3` (from `[python].resolves`). However, it is not compatible with the current targets because:
- The targets use interpreter constraints (`CPython==2.7.* OR CPython==2.7.*,>=3.9`) that are not a subset of those used to generate the lockfile (`CPython>=3.9`).
itās kind of a chicken-egg problem, I cannot add a Python2 compatible package without affecting the rest of my packages (specifically pinning down those package dependencies to be compatible with all my interpreter constraints)?enough-analyst-54434
03/01/2023, 7:58 PM["==2.7.*"]
and the other resolve to have [">=3.9"]
. I'd suggest re-wind to there. And then moving forwards again.[">=2.7,!=3.0.**,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,!=3.8.*"]
.*lively-gpu-26436
03/01/2023, 8:47 PMpants.toml
IC as interpreter_constraints = ["CPython==2.7.*", "CPython>3.6.*"]
2. BUILD
of my python2-python3 packages as:
"interpreter_constraints": [
">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"
],
"resolve": "py2-py3"
1. BUILD
of my python3 only package as:
"interpreter_constraints": [
">=3.11"
],
"resolve": "py3"
I havenāt tested yet but she says itās working. The usage of a python3 only resolve was what I was trying to achieve with the python-default
resolve, but it seems it is affected by the IC defined in pants.toml
enough-analyst-54434
03/01/2023, 8:48 PM"CPython>3.6.*"
is legal, but confusing. It's probably more clear to say "CPython>=3.7"
. Only use *
in ==
constraints."CPython>3.6.*"
-> "CPython>3.6.0"
by the spec actually and that's probably not what you mean.$ pex packaging
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from packaging.specifiers import SpecifierSet
>>> "3.6.1" in SpecifierSet(">=3.6.*")
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/jsirois/.pex/installed_wheels/714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2/packaging-23.0-py3-none-any.whl/packaging/specifiers.py", line 711, in __init__
parsed.add(Specifier(specifier))
File "/home/jsirois/.pex/installed_wheels/714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2/packaging-23.0-py3-none-any.whl/packaging/specifiers.py", line 245, in __init__
raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '>=3.6.*'
>>> "3.6.1" in SpecifierSet("==3.6.*")
True
>>> "3.6.1" in SpecifierSet("==3.7.*")
False
>>> "3.6.1" in SpecifierSet(">3.6")
True
>>> "3.6.1" in SpecifierSet(">=3.7")
False
>>>
now exiting InteractiveConsole...
lively-gpu-26436
03/02/2023, 7:55 AM...
enable_resolves = true
interpreter_constraints = ["CPython==2.7.*", "CPython>=3.7.0"]
[python.resolves]
python-default = "resolves/python-default.lock"
py2-py3 = "resolves/py2-py3.lock"
py3 = "resolves/py3.lock"
[pytest]
lockfile = "resolves/pytest.lock"
version = "pytest==4.6.11"
xdist_enabled = false
extra_requirements = [
"pytest-cov>=2.12,!=2.12.1,<3.1",
]
...
but for some reason generating the lockfiles gets stuck in pytest for me:enough-analyst-54434
03/02/2023, 8:07 AMinterpreter_constraints = ["CPython==2.7.*", "CPython>=3.7.0"]
instead of interpreter_constraints = ["CPython==2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"]
- is there a reason for that?lively-gpu-26436
03/02/2023, 8:08 AMenough-analyst-54434
03/02/2023, 8:08 AMlively-gpu-26436
03/02/2023, 8:22 AMinterpreter_constraints = ["CPython==2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"]
and interestingly is erroring with this09:21:43.48 [ERROR] 1 Exception encountered:
Engine traceback:
in `generate-lockfiles` goal
in Generate Python lockfile
ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:
stderr:
pid 15714 -> /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/48ab48c36a1be9f0243105b49691c65ecbd3b4bd/bin/python -sE /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/48ab48c36a1be9f0243105b49691c65ecbd3b4bd/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-feature 2020-resolver --isolated -q --cache-dir /Users/anler/.cache/pants/named_caches/pex_root/pip_cache --log /private/var/folders/z6/y9s0fw1s0k1fmn_9cxyfllrm0000gn/T/pants-sandbox-BPuFJr/.tmp/pex-pip-log.dobbdhgy/pip.log download --dest /private/var/folders/z6/y9s0fw1s0k1fmn_9cxyfllrm0000gn/T/pants-sandbox-BPuFJr/.tmp/tmph_3epyjc/Users.anler..pyenv.versions.2.7.18.bin.python2.7 aiohttp[speedups] ipdb prometheus-client pyarrow pydantic requests typer types-requests --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
WARNING: Missing build requirements in pyproject.toml for ipdb from <https://files.pythonhosted.org/packages/23/b2/c972cc266b0ba8508b42dab7f5dea1be03ea32213258441bf1b00baca555/ipdb-0.13.11.tar.gz#sha256=c23b6736f01fd4586cc2ecbebdf79a5eb454796853e1cd8f2ed3b7b91d4a3e93>.
WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
WARNING: Discarding <https://files.pythonhosted.org/packages/00/ca/cac34ea9525dd39084a67339fe3018052acf23efb3ce4599c08f6cd21528/pydantic-0.1.tar.gz#sha256=3b57849a1a58af90c3675562d6026d6667c1860adf5bddcce04279a369931654> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/7d/5e/dbc0f68e407b9909542cce19629f3c5ec35a57428cf7db20fbb24daef1df/pydantic-0.0.8.tar.gz#sha256=711a83a58a7812f1c4baeffbe9fc0d11f166703efa6a895830b2c6c2a8563ba5> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/3f/b4/1cd0ce2e0042cfa900b193cb2be9d7c39e96182435368a35355a2fff724d/pydantic-0.0.7.tar.gz#sha256=38a560a2e33c5cb4c990893fe86e5dac0ff2ab9c758989bed8ff4ab69870dc8b> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/ba/05/3f24339fff26bcab7b4e7c4659be71859f5b0f631dded965ac686e8043ba/pydantic-0.0.6.tar.gz#sha256=90ae509b99201af4011e88f57895d653fb9cb9207d6f43e435bb38f9468a5316> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/85/9b/3dab62133e36c33693d22456ab6b503c33a595dce432bc74190ac97e8d75/pydantic-0.0.5.tar.gz#sha256=dbd7ffb60c6544223aaf2aad9d33275be90a419c2d79c6c6967d6237cca12982> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/7c/e7/50e5b6d6ebd4326a9f1d93a5e63d5725c8e88c17b722eda992ad0c09c267/pydantic-0.0.4.tar.gz#sha256=b8b62d94f5ab9cb742024bc6d9ec9f94c089ab0a16089b588828d4a5f72e05b0> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/99/85/a19d6de523cd719a97bd5e917eb723235cd17528a92338d6090e916e9292/pydantic-0.0.3.tar.gz#sha256=b4dfac711fb819a68928e4495658032ab1b3eba3a201d9bc104837ddd4f5d71c> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/94/8a/cbb49f2a5a0302f2a34f782171e6d53360eb90dc1dd9dd8da976e7d25ef6/pydantic-0.0.2.tar.gz#sha256=65362cbb2ae64d97b401cd07560c5a02a536c2046c09f9e99958967f78fda799> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/2b/7e/aae6a7efad925961f27e4af4206b4643614f5ea5fa7e4b5d7739dcf863ff/pydantic-0.0.1.tar.gz#sha256=f9622612da3f7dfc04ba19e0ca2fdd3f32be075fdcb6288794817165bcebc624> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pydantic
ERROR: No matching distribution found for pydantic
[python.resolves_to_interpreter_constraints]
py2-py3 = ["==2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"]
py3 = [">=3.7.0"]
enough-analyst-54434
03/02/2023, 8:26 AMlively-gpu-26436
03/02/2023, 8:29 AMenough-analyst-54434
03/02/2023, 8:30 AMlively-gpu-26436
03/02/2023, 8:31 AMenough-analyst-54434
03/02/2023, 8:32 AMlively-gpu-26436
03/02/2023, 8:33 AM[python]
# ...
interpreter_constraints = ["==3.9.*"]
[python.resolves]
python-default = "resolves/python-default.lock"
py2-py3 = "resolves/py2-py3.lock"
py3 = "resolves/py3.lock"
[python.resolves_to_interpreter_constraints]
python-default = [">=3.7,<3.10"]
py2-py3 = [
">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,<3.10",
]
py3 = [">=3.11"]
src/python
and src/python2-compatible
, and a BUILD file in each of them with the proper defaults for IC and resolve__defaults__(
all={
"interpreter_constraints": [
">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,<3.10",
],
"resolve": "py2-py3",
"skip_flake8": True,
"skip_mypy": True,
"skip_isort": True,
"skip_black": True,
"skip_pylint": True,
}
)