Working with the new resolves, trying to figure ou...
# general
Working with the new resolves, trying to figure out what’s what I have one question and one error report feedback for pex 🧵
First the error feedback issue. I got this when running
./pants package
for a
that depends on a `python_distribution`:
Copy code
10:18:41.98 [INFO] Completed: Extracting 1 requirement to build bin.pex from libs_lockfile.pex: y-cdi-server<7.0.0,>=6.18.10
10:18:41.99 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Extracting 1 requirement to build bin.pex from libs_lockfile.pex: y-cdi-server<7.0.0,>=6.18.10' failed with exit code 1.

Failed to resolve requirements from PEX environment @ /Users/x/.cache/pants/named_caches/pex_root/unzipped_pexes/6fba5552656884e2d57d5e870d4e77659011a422.
Needed cp39-cp39-macosx_12_0_x86_64 compatible dependencies for:
 1: thrift
    Required by:
      FingerprintedDistribution(distribution=y-logging 2.1.1 (/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/1312836c8e4a7e2af38847a85c99385a4409622e6b172a979f55c799f0be6471/y_logging-2.1.1-py3-none-any.whl), fingerprint='979ed43ea397375058082a1e33ace9d24dbb60d8182a8721ce55f2e6e48086d8')
    But this pex had no 'thrift' distributions.
However, the statement that the pex doesn’t have a
distribution is untrue, as with verbose pex output, it says it has this distribution in the pex:
pex:  - thrift-0.15.0-cp38-cp38-macosx_11_0_x86_64.whl
so it doesn’t have any compatible distribution in the pex. Which is my error report feedback.
Which leads me to my question: why does it need the cp39 one ? Here’s the command line for that pex run:
Copy code
/Users/x/.pyenv/versions/3.7.12/bin/python ./pex --tmpdir .tmp --jobs 1 --cert .cacert.pem --python-path $'/Users/x/.pyenv/versions/3.7.12/bin:/Users/x/.pyenv/versions/3.8.11/bin:/Users/x/.pyenv/versions/3.9.7/bin:/usr/local/bin:/root/pyenv/bin:/usr/bin' --output-file bin.pex --no-emit-warnings --manylinux manylinux2014 --include-tools --requirements-pex local_dists.pex --pex-repository libs_lockfile.pex --interpreter-constraint $'CPython>=3.8' --entry-point $'cdi.cli:main' $'--sources-directory=source_files' $'y-cdi-server<7.0.0,>=6.18.10' --layout zipapp
has a
and in my
I have
Copy code
interpreter_constraints = [">=3.8"]
So I’d figure they ought to be compatible, but I got the feeling that pants picks the lowest possible version satisfying the constraints, while pex picks the highest.. is that possible? (will run some experiments wrgt this..)
Hmm.. of all the pythons, it seems it picked the system one, ugh!
Copy code
pex: Constraints on interpreters: (Requirement.parse('CPython>=3.8'),), Matching Interpreter: /Applications/
wait a minute, that’s a 3.8 one, so why does it then complain about?
Copy code
Needed cp39-cp39-macosx_12_0_x86_64 compatible dependencies ...
(sorry for spamming here… using this area to think..)
Didn't Benjy mention here that Pants doesn't guarantee the lowest matching python version;cid=C046T6T9U
👍 1
Hmm, yeah ok. But from the pex logs, it seems that a py3.8 was selected, and yet it finds a dep that requires 3.9.. ah, right, perhaps it was a 3.9 that produced the lock file, thanks!
OK, so pinning my interpreter constraints to
does work. However I don’t think it was the interpreter selected to generate the lockfile that matters (noticed in the diff that I got rid of a lot of artifacts for various versions of python then..) so perhaps it is the one selected when creating the resolve pex that differs.
The springing point being that I’ve only got a thrift wheel for py38, as all the others are either py3-none-any or wheels for both py38 and py39..
Aha! The resolve lockfile.pex is built using
--interpreter-constraint $'CPython<3.9,>=3.8'
I wonder where that constraint came from..?
(starting to feel like it could be beneficial to include the selected python version and/or constraints for the various tasks in the debug log)
Hmm… so under what circumstances is this used?
Copy code
11:47:33.76 [DEBUG] Completed: Find Python interpreter for constraints - Selected /Users/x/.pyenv/versions/3.8.11/bin/python3.8 to run PEXes with.
as most of the pex invocations I see starts with:
Copy code
[DEBUG] spawned local process as Some(74821) for Process { argv: ["/Users/x/.pyenv/versions/3.7.12/bin/python", "./pex", ...
although the building of
did use the 3.8 one, but that was the only pex invocation that did, all the others used 3.7.12
Ok, I think I may have narrowed the issue down, somewhat, hopefully. Sorry for the lengthy issue, @enough-analyst-54434 Could use some expertise on rooting this out all the way.
Oh, btw I didn’t notice that
resolves_to_interpreter_constraints = { libs = [">=3.8,<3.10"] }
had any effect at all when applying ICs to install the resolve for the pex binary. @hundreds-father-404 is it a known limitation that this option doesn’t apply to
targets when using local dists? (if I’ve drawn the correct conclusions)
Conclusion I based on what ICs where used when invoking pex, where I had
interpreter_constraints = [">=3.8,<4"]
set, and I can see that IC a lot, but never
Thanks @curved-television-6568 - that looks legit. I cleared away the cruft with a compact repro on the issue and will continue from there.
❤️ 1