https://pantsbuild.org/ logo
Join Slack
Powered by
Hmm… there’s an issue with using `pants_requiremen...
# plugins
c
Hmm… there’s an issue with using 
pants_requirements()
target on the 2.11.0rc1 release.. 🧵
✅ 1
Copy code
ProcessExecutionFailure: Process 'Generate lockfile for 3rdparty-deps' failed with exit code 1.
stdout:

stderr:
ERROR: Cannot install pantsbuild-pants==2.11.0rc0 and pantsbuild-pants==2.11.0rc1 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
pid 8619 -> /Users/x/.cache/pants/named_caches/pex_root/venvs/d73eb1958462afae0ef27208a6356f1bd127bac9/5227c1516c74d286d844078a1012386b2283a2d5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/x/.cache/pants/named_caches/pex_root --log /private/tmp/tmp9rzue6fg/pip.log download --dest /private/tmp/tmprv074hpp/Users.x..pyenv.versions.3.7.12.bin.python3.7 pantsbuild.pants.testutil<2.12,>=2.11.0rc0 pantsbuild.pants<2.12,>=2.11.0rc0 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
 
 The conflict is caused by:
     pantsbuild-pants 2.11.0rc1 depends on pex==2.1.75
     pantsbuild-pants 2.11.0rc0 depends on pex==2.1.72
 
 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict
Well… I think 
pip
ought to have found a way out by using the 
rc1
release given the desired constraints of 
pantsbuild.pants<2.12,>=2.11.0rc0
… ?
The 
pants.toml
have 
pants_version="2.11.0rc1"
in this case… maybe that is a missing input to all this?
Addressed by bumping the lower end of the constraint: https://github.com/pantsbuild/pants/pull/15043
But hmm… I get the same error also when I use 
pants_version="2.11.0rc0"
in 
pants.toml
..
Hmm… the 
pip
command seemed to run ok, but exits with 1.. ?
Copy code
$ /Users/x/.cache/pants/named_caches/pex_root/venvs/b295182ab4c550677e62422803f6a58e63ee0a33/5227c1516c74d286d844078a1012386b2283a2d5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -vvv --cache-dir /Users/x/.cache/pants/named_caches/pex_root --log ./pip.log download --dest ./pip-down 'pantsbuild.pants.testutil<2.12,>=2.11.0rc0' 'pantsbuild.pants<2.12,>=2.11.0rc0' --index-url <https://pypi.org/simple/> --retries 5 --timeout 15
Ends after a lot of log lines with:
Copy code
[...]
Saved ./pip-down/six-1.16.0-py2.py3-none-any.whl
Saved ./pip-down/setuptools-57.5.0-py3-none-any.whl
Successfully downloaded pantsbuild.pants ansicolors fasteners humbug ijson packaging pex psutil python-lsp-jsonrpc setproctitle toml types-PyYAML types-setuptools types-toml typing-extensions pantsbuild.pants.testutil pyparsing pytest attrs importlib-metadata pluggy py PyYAML tomli ujson zipp iniconfig requests certifi charset-normalizer idna urllib3 six setuptools
Removed build tracker: '/private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/pip-req-tracker-6fl7gsxj'
while the 
__run.sh
script from the sandbox errors out with:
Copy code
ERROR: Cannot install pantsbuild-pants==2.11.0rc0 and pantsbuild-pants==2.11.0rc1 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
pid 23011 -> /Users/x/.cache/pants/named_caches/pex_root/venvs/b295182ab4c550677e62422803f6a58e63ee0a33/5227c1516c74d286d844078a1012386b2283a2d5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -vvv --cache-dir /Users/x/.cache/pants/named_caches/pex_root --log /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/tmpxhku4u0l/pip.log download --dest /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/tmpe9f2ex1e/Users.x..pyenv.versions.3.7.12.bin.python3.7 pantsbuild.pants.testutil<2.12,>=2.11.0rc0 pantsbuild.pants<2.12,>=2.11.0rc0 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
 
 The conflict is caused by:
     pantsbuild-pants 2.11.0rc1 depends on pex==2.1.75
     pantsbuild-pants 2.11.0rc0 depends on pex==2.1.72
which has the pip command I attempted to run directly, but it didn’t complain about conflicts then, so I must be missing some pieces here..
Huh… not sure if this is relevant, or even the root cause, but when I try to pin to 2.11.0rc0 I get another issue:
Copy code
ERROR: Could not find a version that satisfies the requirement pex==2.1.72
ERROR: No matching distribution found for pex==2.1.72
I can pip install 
pex==2.1.72
just fine…
There’s something seriously fishy going on here, and I’m utterly confused:
Copy code
ERROR: Cannot install pantsbuild-pants==2.11.0rc1 and pex because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
pid 25913 -> /Users/x/.cache/pants/named_caches/pex_root/venvs/d73eb1958462afae0ef27208a6356f1bd127bac9/1f60713df5937a496dd6818c57bcdefa31df4883/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/x/.cache/pants/named_caches/pex_root --log /private/tmp/tmpezlq1q3u/pip.log download --dest /private/tmp/tmpe4ushdeg/Users.x..pyenv.versions.3.8.11.bin.python3.8 pantsbuild.pants.testutil==2.11.0rc1 pantsbuild.pants==2.11.0rc1 pex --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
 
 The conflict is caused by:
     The user requested pex
     pantsbuild-pants 2.11.0rc1 depends on pex==2.1.75
Here, I’ve uninstalled py3.7, just to make sure it wasn’t related to the python version. And I’ve moved away from 
pants_requirements
and instead added them to my 
requirements.txt
file, which only has these lines in it, for an otherwise bare project:
Copy code
pantsbuild.pants==2.11.0rc1
pantsbuild.pants.testutil==2.11.0rc1
pex
Oh… looking at another project where I’ve managed to use the “pex” resolver, I noticed I filed this issue https://github.com/pantsbuild/pex/issues/1688 Using the same ICs here (i.e. 
">=3.8,<3.9"
) solved it. Probably for other reasons, but I’ve no clue as to why… 😉
e
There is a lot of self-debugging here that seems to terminate in continued confusion; yet the thread is marked done. @curved-television-6568 should I pay attention to all these debugging details or are you all set?
c
Uh, it’s @bitter-ability-32190 that’s marked it as done.. ??
I ended up being able to go forward after pinning my ICs to a 3.8.* python
b
My bad I thought the message above meant you solved it
👍 1
c
so I feel there’s an issue here that I don’t understand
e
Ok. I'll take a look this am.
🙏 1
@curved-television-6568 it may or may not be relevant, but I see this is mac (
/Users/
) - are you m1?
And, batching up questions, what does this give you? (Use --python <path to your pyenv 3.7.12 interpreter for apples-to-apples in your OP 1st example log):
Copy code
pex --python python3.7 "pantsbuild.pants.testutil<2.12,>=2.11.0rc0" "pantsbuild.pants<2.12,>=2.11.0rc0" --resolver-version pip-2020-resolver -o kaos.pex
That works for me and selects:
Copy code
$ pex-tools ./kaos.legacy.pex info | jq -r '.distributions | keys[]' | grep -E "pex|pantsbuild"
pantsbuild.pants-2.11.0rc1-cp37-cp37m-manylinux2014_x86_64.whl
pantsbuild.pants.testutil-2.11.0rc1-py37.py38.py39-none-any.whl
pex-2.1.75-py2.py3-none-any.whl
And - last one - if you have the 
__run.sh
script still available from:
while the 
__run.sh
script from the sandbox errors out with:
That would be interesting to see along with knowing which Process it is associated with if you have the relevant output log lines that reveal that.
This is all pretty chaotic to read through and there are good tidbits but also lots of missing log output that might help. Consider filing an issue with this all a bit more organized if that makes it easier for you.
Aha, sorry - missed context. This is when creating a lock. So the pip download command in your log output is from a lock generation session; so the appropriate command to run is:
Copy code
pex3 lock create "pantsbuild.pants.testutil<2.12,>=2.11.0rc0" "pantsbuild.pants<2.12,>=2.11.0rc0" --resolver-version pip-2020-resolver -o lock.json --indent 2 --style universal
That works for me still, but I may have enough to go on to gedanken this...
Yeah, no, not enough. @curved-television-6568 the more full context you could give, the better, preferably in an issue. The actual commands run along with un-snipped log output if possible.
👍 1
c
Will do, along with trying the above commands (likely tomorrow). Thanks for looking into this.
(on a Mac, but not M1)
MacBook Pro (15-inch, 2018), v12.3.1, 2,6 GHz 6-Core Intel Core i7
e
Great - thank you.
c
I checked my steps on a Ubuntu 21.10 vm, and it repros there as well, so I’ve posted what I’ve got so far. Will add more logs etc to it tomorrow: https://github.com/pantsbuild/pants/issues/15058
e
Great - thank you. A repro-able image is gold.
Aha! Cookiecutter. The gods are hurling thunderbolts because of that. I suspect I can find a better explanation though.
🍪 1
🌩️ 2
Ah - yeah, of course. Pex was telling you the truth.
Good luck building a pants plugin with Python <4 right?
🎯 1
>=3.8,<3.10
and it works absolutely fine.
c
Aahh… right. OK. That took me 2/3 of my morning dog-walk to figure out why, and then 15 minutes at my desk to confirm. Thanks for just leaving the hints and have me draw the conclusions, very educative 😄 👍
So, the culprit being this line from the pex dist:
Copy code
Requires-Python: >=2.7,<3.11,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*
So, it’s all very obvious “in my face” kind of, once I’ve managed to connect the dots. The error message I got however, doesn’t really help with that, and more so when the pants version was not pinned down, the error message was even more confusing.
I’m going to bite down on this one, as I feel this is a very easy mistake to make, and that you can end up here also with a more narrow constraint, as all it takes is one, possibly transient, dependency to have an incompatible IC from what you requested and boom. Being tricky and hard to track down, here should be plenty of room for improvement.
e
We'll the culprit is right on the tin. Pants itself only publishes 3.7-3.9 wheels and no sdist. You need look no further.
So this wasn't me trying to be educative actually! It's really obvious (after the fact)!!
The pantsbuild.pants dist is very odd in this regard. I can think of no other that is simply completely unavailable unless you have exactly a small set of pythons that are not even a small set of the latest - 3.10 made Pants really stick out.
c
I agree this is all very obvious, once it’s dawned. Are you saying that there’s no risk to run across some odd library that possibly haven’t been updated for a while that doesn’t have a 
Requires-Python: …, <3.8
in it?
On that note, shouldn’t the pants dist have a 
Requires-Python
in it’s METADATA then?
e
It can't be that obvious since you thought I meant Pex was the issue!
c
I got my example working with 
>=3.8,<3.11
too…
e
Hrm, that is surprising.
c
Oh, ok, no not that obvious then. The obvious part I was referring to was that the issue was with the ICs… 😉
e
Ah.
I saying the obvious thing as a Pants dev is it should never work once you mix in Pythons 3.6 or lower or 3.10 or higher.
c
I figured it works with 
<3.11
as pex is ok with 3.10, and pants doesn’t have any constraints.
e
The fact you got it working here in a range that includes 3.10 is a latent bug.
c
Gotcha.
e
At runtime that --lock will fail for 3.10.
c
Glad I found a bug and not only confusion, then 😄
e
It's not a fixable big mind you.
c
No? wouldn’t adding the 
Requires-Python
to the pants dist METADATA solve this?
e
We'll, sure, but how many projects are amenable to that fix? I mean the Pex universal lock generation is not fixable here generally afaict. It requires a friendly project to fix its metadata.
c
Ah, right, yeah the bug I was referring to was that we get a broken lock for pants specifically here.. not solving the issue with broken locks for dists that doesn’t provide proper metadata.
e
So, that will fix this case, definitely. This sort of confusion will still be there though generally, ready to pounce, with no solution.
c
indeed. So I’ll invest some effort into improving the pip error messages to more helpful in that area
e
PyPA simply offers too many redundant degrees of freedom in packaging metadata.
c
So perhaps the error message can’t be more precise, but it can be expanded to help point you in the general direction of the issue.
e
The Pip thing will be stupidly hard afaict, but excellent if you can work up something.
❤️ 1
c
I see now that the message actually hints at the issue by saying “no matching distribution found” here I didn’t catch the significance of “matching”.
and what it didn’t match.
Thank you John, really appreciate the help shredding the mist of confusion I was under 🙂
e
Yeah, I'm super skeptical. You can always toss in a specific case that bit you as a maybe in an error message, but it will only apply to 1% of errors and just inject more goose chase for the other 99% of errors. That's the challenge.
c
Yeah, I’ll look at this issue with some “genericality” goggles on.
But I think pip must be able to provide more details to why a certain dist was deemed unusable for a given requirement.
e
Good, thank you. I'm notoriously on the side of not lying which will be hard here. But maybe there is some middle ground that isn't just pure noise.
➕ 2
c
I appreciate that. It’s like in traffic. I urge drivers to use their directional lights when turning, but it is worse when they do, in the wrong direction.
Hmm… you say that the ICs aren’t part of the pip call… then how come pip fails?
I’ll do more investigations on my end…
e
Pex turns ICs into two contiguous lists of version numbers, a list of major.minor covering the range and a list of major.minor.patch covering the range. It then runtime patches a few strategic functions where Pip delegates evaluation of things like wheel tag applicability and Requires-Python applicability.
This is only for the universal case. For strict locks Pip just operates normally.
c
Aha! So, I could, at least in theory, keep track of rejected versions based on those runtime patches, for the universal case, and at the end if the resolve fails, point out at some level of detail what was rejected and why.
But, from the comment on GH, I think that the route of having Pants augment error messages feels more feasible, for the general case any way.
e
Perhaps. One thing to balance is this is tricky code that forms our ~one non-CLI API binding to Pip. For each one of these bindings, upgrading vendored Pip gets harder. As such a generic goal is to do as little as possible runtime patching / have as little as possible dependence on the Pip logs (which is where most lock info comes from).
c
Agreed. I’d prefer to not mud the waters in those areas.
e
The areas, FWIW are the Locker class in pex/pip/tool.py and the runtime patches to pip are in pex/pip/runtime_patches.py
👌 1
6 Views
Open in Slack
PreviousNext
Powered by