https://pantsbuild.org/ logo
Powered by
l

lively-gpu-26436

03/01/2023, 7:27 PM
Hi all again šŸ˜… Iā€™m having this super weird issue, I had my monorepo configured with a defaUlt interpreter constraints 
["==3.11.*"]
in 
pants.toml
and a default resolve with some dependencies, all worked perfectly here, the issue comes when adding a py2/py3 library, I changed the interpreter constraints in 
pants.toml
to 
["==2.7.*", ">=3.9"]
and used it as the interpreter constraints for my newly added package too, created a new resolve called 
py2-py3 = resolves/py2-py3.lock
and re-generated lock files. The problem comes when running the 
check
goal on my previously working python3 only code, it takes quite a lot of time and finally errors. My question is if someone knows how adding a package with mixed interpreter constraints can cause this check goal to fail in other packages with different constraints that were previously working?
Copy code
./pants check src/python::
20:19:50.13 [INFO] Completed: Building 4 requirements for requirements.pex from the resolves/python-default.lock resolve: aiohttp[speedups], prometheus-client, pyarrow, pydantic
20:19:50.13 [ERROR] 1 Exception encountered:

Engine traceback:
  in `check` goal
  in Typecheck using MyPy - mypy

ProcessExecutionFailure: Process 'Building 4 requirements for requirements.pex from the resolves/python-default.lock resolve: aiohttp[speedups], prometheus-client, pyarrow, pydantic' failed with exit code 1.
stdout:

stderr:
There was 1 error downloading required artifacts:
1. pyarrow 0.16 from <https://files.pythonhosted.org/packages/3e/00/4e099e2f1992c555771090b5971c59751fc152c68270a31bfeabd114c068/pyarrow-0.16.0.tar.gz>
    ERROR: Command errored out with exit status 1: /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/791e686d7c6a49493bab896fbb2645175e97bb31/bin/python /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/791e686d7c6a49493bab896fbb2645175e97bb31/lib/python3.11/site-packages/pip install --ignore-installed --no-user --prefix /Users/anler/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-build-env-fa06xvk8/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i <https://pypi.org/simple/> -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel Check the logs for full command output.

Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
itā€™s like my previous requirements Are trying to be satisfied with the new interpreter constraints, I check if 
python_requirements
target accepts an interpreter_constraints key but it doesnā€™t
I guess this comes down to the question: does my 
pants.toml
interpreter_constraints be a superset of the ones used by my library? If I try to keep my default interpreter_constraints in 
pants.toml
to say 
>=3.9
, then if I try to run the tests of my python2/3 package which is defined as:
Copy code
python_tests(
    name="tests",
    resolve="py2-py3",
    interpreter_constraints=parametrize(py2=["==2.7.*"], py3=[">=3.9"]),
)
then it errors with:
Copy code
InvalidLockfileError: You are using the lockfile at resolves/py2-py3.lock to install the resolve `py2-py3` (from `[python].resolves`). However, it is not compatible with the current targets because:

- The targets use interpreter constraints (`CPython==2.7.* OR CPython==2.7.*,>=3.9`) that are not a subset of those used to generate the lockfile (`CPython>=3.9`).
itā€™s kind of a chicken-egg problem, I cannot add a Python2 compatible package without affecting the rest of my packages (specifically pinning down those package dependencies to be compatible with all my interpreter constraints)?
e

enough-analyst-54434

03/01/2023, 7:58 PM
So, we allow for interpreter constraints to be a list, but you should never use that feature. The OR mixing with separate resolves is not good and the OR is not standard - the Python ecosystem does not support it. So you just want one resolve to have the IC of 
["==2.7.*"]
and the other resolve to have 
[">=3.9"]
. I'd suggest re-wind to there. And then moving forwards again.
If you want to stick with 1 resolve, which may not be possible, you'd use a default IC of 
[">=2.7,!=3.0.**,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,!=3.7.*,!=3.8.*"]
.*
That's the same meaning as your OR without the OR.
You never need an OR. You may just need to type alot of excludes!
But it's PyPA standard that way.
So, basically, 2 things to try: + 1 py2-py3 resolve, parameterized ICs + 1 py2 resolve, 1 py3 resolve, parameterized resolves I'm not sure which will work, but either way - no IC ORs needed.
If you get really stuck, this is deep enough in the weeds that an example repo would be very useful.
l

lively-gpu-26436

03/01/2023, 8:47 PM
Thanks @enough-analyst-54434 I really appreciate the effort. One colleague just did something similar of what youā€™ve told me. So weā€™ve ended up with: 1. 
pants.toml
IC as 
interpreter_constraints = ["CPython==2.7.*", "CPython>3.6.*"]
2. 
BUILD
of my python2-python3 packages as:
Copy code
"interpreter_constraints": [
    ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"
 ],
"resolve": "py2-py3"
1. 
BUILD
of my python3 only package as:
Copy code
"interpreter_constraints": [
    ">=3.11"
 ],
"resolve": "py3"
I havenā€™t tested yet but she says itā€™s working. The usage of a python3 only resolve was what I was trying to achieve with the 
python-default
resolve, but it seems it is affected by the IC defined in 
pants.toml
e

enough-analyst-54434

03/01/2023, 8:48 PM
I don't like #1! #2 looks better
Also, 
"CPython>3.6.*"
is legal, but confusing. It's probably more clear to say 
"CPython>=3.7"
. Only use 
*
in 
==
constraints.
I think 
"CPython>3.6.*"
-> 
"CPython>3.6.0"
by the spec actually and that's probably not what you mean.
šŸ‘ 1
Yeah, not even sure how the heck that is working for you. Here is the PyPA authority on all this (https://pypi.org/project/packaging/):
Copy code
$ pex packaging
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from packaging.specifiers import SpecifierSet
>>> "3.6.1" in SpecifierSet(">=3.6.*")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/jsirois/.pex/installed_wheels/714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2/packaging-23.0-py3-none-any.whl/packaging/specifiers.py", line 711, in __init__
    parsed.add(Specifier(specifier))
  File "/home/jsirois/.pex/installed_wheels/714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2/packaging-23.0-py3-none-any.whl/packaging/specifiers.py", line 245, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '>=3.6.*'
>>> "3.6.1" in SpecifierSet("==3.6.*")
True
>>> "3.6.1" in SpecifierSet("==3.7.*")
False
>>> "3.6.1" in SpecifierSet(">3.6")
True
>>> "3.6.1" in SpecifierSet(">=3.7")
False
>>>
now exiting InteractiveConsole...
l

lively-gpu-26436

03/02/2023, 7:55 AM
this is how it looks at the end with your tips
Copy code
...
enable_resolves = true
interpreter_constraints = ["CPython==2.7.*", "CPython>=3.7.0"]

[python.resolves]
python-default = "resolves/python-default.lock"
py2-py3 = "resolves/py2-py3.lock"
py3 = "resolves/py3.lock"

[pytest]
lockfile = "resolves/pytest.lock"
version = "pytest==4.6.11"
xdist_enabled = false
extra_requirements = [
  "pytest-cov>=2.12,!=2.12.1,<3.1",
]
...
but for some reason generating the lockfiles gets stuck in pytest for me:
e

enough-analyst-54434

03/02/2023, 8:07 AM
@lively-gpu-26436 you're still using 
interpreter_constraints = ["CPython==2.7.*", "CPython>=3.7.0"]
instead of 
interpreter_constraints = ["CPython==2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"]
- is there a reason for that?
l

lively-gpu-26436

03/02/2023, 8:08 AM
arrg sorry I thought that syntax was only for the IC in BUILD files, going to change it now
e

enough-analyst-54434

03/02/2023, 8:08 AM
It's the universal IC syntax - same everywhere.
āœ… 1
So you have 3 resolves but you don't show 3 different ICs - I would have expected you'd use a custom IC per resolve.
l

lively-gpu-26436

03/02/2023, 8:22 AM
whoa was looking for something like that yesterday but failed miserably, I need to get better at navigating the docs
I tried with 
interpreter_constraints = ["CPython==2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"]
and interestingly is erroring with this
Copy code
09:21:43.48 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal
  in Generate Python lockfile

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 15714 -> /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/48ab48c36a1be9f0243105b49691c65ecbd3b4bd/bin/python -sE /Users/anler/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/48ab48c36a1be9f0243105b49691c65ecbd3b4bd/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-feature 2020-resolver --isolated -q --cache-dir /Users/anler/.cache/pants/named_caches/pex_root/pip_cache --log /private/var/folders/z6/y9s0fw1s0k1fmn_9cxyfllrm0000gn/T/pants-sandbox-BPuFJr/.tmp/pex-pip-log.dobbdhgy/pip.log download --dest /private/var/folders/z6/y9s0fw1s0k1fmn_9cxyfllrm0000gn/T/pants-sandbox-BPuFJr/.tmp/tmph_3epyjc/Users.anler..pyenv.versions.2.7.18.bin.python2.7 aiohttp[speedups] ipdb prometheus-client pyarrow pydantic requests typer types-requests --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  WARNING: Missing build requirements in pyproject.toml for ipdb from <https://files.pythonhosted.org/packages/23/b2/c972cc266b0ba8508b42dab7f5dea1be03ea32213258441bf1b00baca555/ipdb-0.13.11.tar.gz#sha256=c23b6736f01fd4586cc2ecbebdf79a5eb454796853e1cd8f2ed3b7b91d4a3e93>.
  WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
WARNING: Discarding <https://files.pythonhosted.org/packages/00/ca/cac34ea9525dd39084a67339fe3018052acf23efb3ce4599c08f6cd21528/pydantic-0.1.tar.gz#sha256=3b57849a1a58af90c3675562d6026d6667c1860adf5bddcce04279a369931654> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/7d/5e/dbc0f68e407b9909542cce19629f3c5ec35a57428cf7db20fbb24daef1df/pydantic-0.0.8.tar.gz#sha256=711a83a58a7812f1c4baeffbe9fc0d11f166703efa6a895830b2c6c2a8563ba5> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/3f/b4/1cd0ce2e0042cfa900b193cb2be9d7c39e96182435368a35355a2fff724d/pydantic-0.0.7.tar.gz#sha256=38a560a2e33c5cb4c990893fe86e5dac0ff2ab9c758989bed8ff4ab69870dc8b> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/ba/05/3f24339fff26bcab7b4e7c4659be71859f5b0f631dded965ac686e8043ba/pydantic-0.0.6.tar.gz#sha256=90ae509b99201af4011e88f57895d653fb9cb9207d6f43e435bb38f9468a5316> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/85/9b/3dab62133e36c33693d22456ab6b503c33a595dce432bc74190ac97e8d75/pydantic-0.0.5.tar.gz#sha256=dbd7ffb60c6544223aaf2aad9d33275be90a419c2d79c6c6967d6237cca12982> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/7c/e7/50e5b6d6ebd4326a9f1d93a5e63d5725c8e88c17b722eda992ad0c09c267/pydantic-0.0.4.tar.gz#sha256=b8b62d94f5ab9cb742024bc6d9ec9f94c089ab0a16089b588828d4a5f72e05b0> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/99/85/a19d6de523cd719a97bd5e917eb723235cd17528a92338d6090e916e9292/pydantic-0.0.3.tar.gz#sha256=b4dfac711fb819a68928e4495658032ab1b3eba3a201d9bc104837ddd4f5d71c> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/94/8a/cbb49f2a5a0302f2a34f782171e6d53360eb90dc1dd9dd8da976e7d25ef6/pydantic-0.0.2.tar.gz#sha256=65362cbb2ae64d97b401cd07560c5a02a536c2046c09f9e99958967f78fda799> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/2b/7e/aae6a7efad925961f27e4af4206b4643614f5ea5fa7e4b5d7739dcf863ff/pydantic-0.0.1.tar.gz#sha256=f9622612da3f7dfc04ba19e0ca2fdd3f32be075fdcb6288794817165bcebc624> (from <https://pypi.org/simple/pydantic/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pydantic
ERROR: No matching distribution found for pydantic
Iā€™m also pinning each resolve to a IC
Copy code
[python.resolves_to_interpreter_constraints]
py2-py3 = ["==2.7.*,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"]
py3 = [">=3.7.0"]
e

enough-analyst-54434

03/02/2023, 8:26 AM
It looks like pydantic may be Python 3 only.
In their older versions they do not have Requires-Python metadata shown in project details on PyPI, but the wheel implies it: https://pypi.org/project/pydantic/0.0.1/#files
l

lively-gpu-26436

03/02/2023, 8:29 AM
hmm the thing is that Pydantic is only used in a python3 package of my monorepo, and thatā€™s kind of what I want to achieve keep the python3 only packages up to date to the latest and greatest and for just this python2 package I have which doesnā€™t use pydantic use its own set of outdated dependencies
e

enough-analyst-54434

03/02/2023, 8:30 AM
So that's where you need to use resolves. The pydantic requirement should only be in py3 resolves.
A lock will fail unless it can lock all transitive requirements for all interpreters implied by the IC.
l

lively-gpu-26436

03/02/2023, 8:31 AM
oh ok, yes I was missing a resolve=ā€œpy3ā€ in my python_requirements target
e

enough-analyst-54434

03/02/2023, 8:32 AM
I'm sure it won't be fun to get this sorted, but I think you have all the tools you need.
l

lively-gpu-26436

03/02/2023, 8:33 AM
yes, Iā€™m actually impressed that for every corner case I run into thereā€™s a link to Pants docs šŸ˜„
I confirm everything is working now @enough-analyst-54434 thanks a million times! I ended up this `pants.toml`:
Copy code
[python]
# ...
interpreter_constraints = ["==3.9.*"]

[python.resolves]
python-default = "resolves/python-default.lock"
py2-py3 = "resolves/py2-py3.lock"
py3 = "resolves/py3.lock"

[python.resolves_to_interpreter_constraints]
python-default = [">=3.7,<3.10"]
py2-py3 = [
  ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,<3.10",
]
py3 = [">=3.11"]
and separated my sources in 
src/python
and 
src/python2-compatible
, and a BUILD file in each of them with the proper defaults for IC and resolve
the python2-compatible looks like this:
Copy code
__defaults__(
    all={
        "interpreter_constraints": [
            ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,<3.10",
        ],
        "resolve": "py2-py3",
        "skip_flake8": True,
        "skip_mypy": True,
        "skip_isort": True,
        "skip_black": True,
        "skip_pylint": True,
    }
)
7 Views
Powered by