https://pantsbuild.org/ logo
Join Slack
Powered by
We are unable to generate a pex lockfile for our r...
# general
w
We are unable to generate a pex lockfile for our repository 😞
Copy code
22:23:47.42 [INFO] Preserving local process execution dir /tmp/pants-sandbox-RlFtMs for Generate lockfile for python-default
22:23:47.42 [INFO] Completed: Generate lockfile for python-default
22:23:47.42 [DEBUG] Completed: Scheduling: Generate lockfile for python-default
22:23:47.42 [DEBUG] Completed: Generate Python lockfile
22:23:47.42 [DEBUG] Completed: `generate-lockfiles` goal
22:23:47.42 [DEBUG] computed 1 nodes in 111.510922 seconds. there are 15428 total nodes.
22:23:47.42 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.generate_lockfiles.generate_lockfiles_goal
  in pants.backend.python.goals.lockfile.generate_lockfile
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/tobiasn/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pex: Resolving interpreters
pex: Constraints on interpreters: (Requirement(name='CPython', url=None, extras=frozenset(), specifier=<SpecifierSet('~=3.10')>, marker=None),), Matching Interpreter: /home/tobiasn/.pyenv/versions/3.10.2/bin/python3.10
pex: Resolving interpreters :: Resolving for:
  /home/tobiasn/.pyenv/versions/3.10.2/bin/python3.10
pex: Spawning a maximum of 16 parallel jobs to process:
  /home/tobiasn/.pyenv/versions/3.10.2/bin/python3.10
pex: Calculating compatible python versions for ('~=3.10',)
pex: Calculating compatible python versions for ('~=3.10',): 6.7ms
pex: Hashing pex
pex: Hashing pex: 20.0ms
pex: Isolating pex
pex: Isolating pex: 0.0ms
pid 79672 -> /home/tobiasn/.cache/pants/named_caches/pex_root/venvs/ea2d19805769311fbd5354560327cca595da53cd/b4fdfcd2e0324f57abb08ab22409e095be4739d7/bin/python -sE /home/tobiasn/.cache/pants/named_caches/pex_root/venvs/ea2d19805769311fbd5354560327cca595da53cd/b4fdfcd2e0324f57abb08ab22409e095be4739d7/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -vvv --cache-dir /home/tobiasn/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-RlFtMs/.tmp/pex-pip-log.x7y7n218/pip.log download --dest /tmp/pants-sandbox-RlFtMs/.tmp/tmp7x_hy3im/home.tobiasn..pyenv.versions.3.10.2.bin.python3.10 --requirement __pip_args.txt *** --index-url <https://aws>:***/pypi/platform-dependencies/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement numpy==1.21.4
ERROR: No matching distribution found for numpy==1.21.4
numpy==1.21.4 is for sure available in
Copy code
<https://aws>:***/pypi/platform-dependencies/simple/
pants version 2.13.0, probably relevant
h
This may be because no wheel is available for a platform+python version you’re targeting
What is in the 
[python]
section of your pants.toml?
If you’re able to create a small dummy repo on github that reproduces the problem, we can diagnose more easily
w
Copy code
[python]
interpreter_constraints = ["CPython~=3.10"]
tailor_pex_binary_targets = false
enable_resolves = true

[python.resolves]
python-default = "pants-lockfile.txt"
We only target 
"linux-x86_64-cp-3.10.2-cp310"
in our 
pex_binaries
afaict, and all packages do have wheels for this platform in our software mirror
Not sure what a repro would look like, and at any rate https://pypi.org/project/numpy/1.21.4/#files contains the numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (and so does our mirror)
is there no way I can at least get more logs from the pex lock thingy? right now it is silent until the cryptic ERROR
./pants -ldebug --dynamic-ui=false --keep-sandboxes=on_failure --pex-verbosity=9 --print-stacktrace generate-lockfiles
is silent once the pex starts the lock-thingy
h
If you run 
./pants --keep-sandboxes=on_failure ...
it’ll print out the location of the sandbox of the failed process
and there will be a 
__run.sh
script in there with the full cmd line the process is running
and you can modify that cmd line and run it in the sandbox dir to try things out
one thing to try might be to trim all the input requirements down to just the 
numpy
one, to isolate the problem
That cmd line might be a helpful repro, as it isolates the problem to just pex, rather than pants+pex
w
Right, so https://github.com/tobni/dummy-pants-repro reproduces it, but I still dont understand why
Copy code
docker run python:3.10-slim pip install numpy==1.21.4
matches the wheel 
numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
, so howcome 
pex
wont have it 😕
Ah, managed to find the issue by adding 
--preserve-pip-download-log
to the 
__run.sh
script
Copy code
2022-09-25T00:22:16,662   Link requires a different Python (3.10.2 not in: '>=3.7,<3.11'): <https://files.pythonhosted.org/packages/ab/94/2fc54cc791846812318080a4f86f9afcf661891163779684d1dd1fe018f9/numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=615d4e328af7204c13ae3d4df7615a13ff60a49cb0d9106fde07f541207883ca> (from <https://pypi.org/simple/numpy/>) (requires-python:>=3.7,<3.11)
So, patch version number matching is seemingly scuffed?
Hmm no, it's
Copy code
interpreter_constraints = ["CPython~=3.10"]
actually causing the issue, changing it to
Copy code
interpreter_constraints = ["CPython==3.10.2"]
resolves it
Funny how I haven't noticed that until now, I'm fairly sure we would not be okay with ANY python minor version higher than 3.10... Still odd that pex thinks 3.10.2 is not in the 3.9-3.11 range, but at least I got past this head scratcher, thanks 🙏
h
Huh...
Thanks for the repro, I'll poke at it
That seems like a gotcha
What happens if you spell it out as 
CPython>=3.10,<3.11
Instead of the ~= operator
w
That also works
But, unless I'm mistaken, 
CPython~=3.10
is actually equivalent to 
CPython>=3.10,<4.0
, and that does error
Your suggestion (
CPython>=3.10,<3.11
) would be the equivalent of 
CPython~=3.10.0
e
I think Benjy's only point was to recreate the range not using ~= syntax. The suspicion is Pex handles that operator incorrectly.
👍 1
h
Yes, I wanted to confirm that it is an issue with Pex handling the 
~=
operator itself, and not something deeper. But also since I never use 
~=
I was misinterpreting its meaning
If 
CPython~=3.10
is equivalent to 
CPython>=3.10,<4.0
then Pex may be doing the right thing here
If, e.g., there are no wheels for 3.11
The lockfile wants to be compatible with all interpreters that satisfy the constraints
This is why it’s generally best to constrain to one or two minor versions
e
Aha, yes. Benjy is totally right. Pex and Pip were doing the right thing since there are no 3.11 wheels and Pex shipped 3.11 compatibility a while back: https://peps.python.org/pep-0440/#compatible-release One could argue Pex should not consider 3.11 real until October when the 1st stable release lands. That probably makes sense. Nonetheless, Benjy's suggestion is wise when you're asking Pex to perform a universal lock. Now you aren't doing that directly, Pants is on your behalf without necessarily fully explaining what that entails. I have some work to do on that front adding docs about locking over in Pex. (Pex IC handling: https://github.com/pantsbuild/pex/blob/1be03fedebbb348f79f9faeb792c7444202f492c/pex/interpreter_constraints.py#L145-L216)
h
Pants docs should probably be a lot more emphatic about not having open-ended interpreter constraints
w
Id argue a better error message would be enough, no? There is no indication 3.11 is the target during failure. Maybe thats harder to accomplish.
Ive just been spoiled by the error messages from pants 🙈
(Notice how the pip log is incorrectly stating 3.10.2 is not in a valid range >=3.7,<3.11)
e
Yeah, I think there is a bugfix to be had still with ~= handling re the 3.10.2. I also agree errors need some treatment. Its certainly not easy - Pex is executing Pip in a subprocess since that's the only supported API; so error correlation will have to be from stderr / stdout compared against the Pip verbose logs and the meaningful bits extracted.
@worried-painter-31382 https://github.com/tobni/dummy-pants-repro appears to be gone. Any chance you could resurrect that? I'd like to use it for a repro case / issue file over in Pex to fix up ~= handling.
w
@enough-analyst-54434 Sorry for getting back so late, here's the spiritual equivalent: https://github.com/tobni/dummy-pants-repo
e
Ok, great, Thank you. I'll take a look and get a bug filed here later today.
Ok: https://github.com/pantsbuild/pex/issues/1927 It turns out `~=3.10`means `>=3.10,==3.10.*`and not `>=3.10,<4`so the lock failure is a straight up bug (I think you knew this but I did not - I was very ignorant of the `~=`meaning which probably explains why I wrote code with bugs handling it~). Using 
>=3.10,==3.10.*
or `>=3.10,<3.11`both work and are exactly equivalent; so this should be fixed up.
Damn - got that wrong - again! 
~=3.10
does mean 
>=3.10,==3.*
aka `>=3.10,<4`and so the lock is definitely correct to fail and this is just an error message problem.
👍 1
7 Views
Open in Slack
PreviousNext
Powered by