Hi I m having some trouble with generate lockfiles hanging f

Question

Hi I m having some trouble with generate lockfiles hanging forever and leaving a process using 100 of a core in the background after I cancel the goal Is there something I could be doing wrong to cause this or should I raise a bug

enough-analyst-54434 · Answer

Can you please provide both ` pants V` and the full output of ` pants ldebug generate lockfiles`

average-flag-94768 · Answer

Untitled txt

average-flag-94768 · Answer

I ve used a few different version 2 12 14 of pants to try to get it to work but currently its 2 14 0 dev5 the debug contained a api key for a private cheeseshop I had to remove

average-flag-94768 · Answer

I also get the same issue on two machines one under wsl and one on ubuntu

enough-analyst-54434 · Answer

Thanks for the details I m a bit confused The log above ends with lock completed lines Are you unhappy with the ~85 seconds it took

average-flag-94768 · Answer

There are two resolves one finishes the other hangs forever

enough-analyst-54434 · Answer

Aha the `api` one Ok

enough-analyst-54434 · Answer

Can you confirm you said it above but to be sure that the `api` resolve hang also happens on non WSL We went through the exercise yesterday of learning WSL mounts windows drives using 9p which is a network filesystem that does not support filesystem events and perhaps does not support file locks If it doesn t support file locks then that would foil Pex which is doing the locking here and could present as a hang

average-flag-94768 · Answer

Yeah I noticed it first on ubuntu and then recreated it on WSL later on

enough-analyst-54434 · Answer

And it was Ubuntu on the metal I E The Pants project was on an ext4 or similar filesystem

average-flag-94768 · Answer

Well zfs but yeah on the metal

average-flag-94768 · Answer

Although ZFS can be a bit weird so maybe that is something

enough-analyst-54434 · Answer

ZFS is a local filesystem so it almost certainly supports POSIX locks Ok Let me try your two resolves there in the debug output It s not clear which is `api` and which is `alert `

average-flag-94768 · Answer

I can get you a log of just the failing one if it makes it easier

enough-analyst-54434 · Answer

That would be great actually

average-flag-94768 · Answer

Untitled txt

enough-analyst-54434 · Answer

Ok great that s the one I had started I had to comment out `proto bindings==0 1 0` since that s on your private PyPI but then I get a hang + CPU repro

enough-analyst-54434 · Answer

Pex should be logging these lines from Pip but is not ```$ grep pip is looking at multiple versions of tmp tmpstuakkbupex pip log pip log 2022 08 18T07 30 06 421 INFO pip is looking at multiple versions of click repl to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 06 421 INFO pip is looking at multiple versions of click plugins to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 06 421 INFO pip is looking at multiple versions of click didyoumean to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 06 611 INFO pip is looking at multiple versions of click to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 06 743 INFO pip is looking at multiple versions of click repl to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 06 744 INFO pip is looking at multiple versions of click plugins to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 06 744 INFO pip is looking at multiple versions of click didyoumean to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 07 200 INFO pip is looking at multiple versions of cheroot to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 07 544 INFO pip is looking at multiple versions of click to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 12 347 INFO pip is looking at multiple versions of cheroot to determine which version is compatible with other requirements This could take a while 2022 08 18T07 30 16 008 INFO pip is looking at multiple versions of charset normalizer to determine which version is compatible with other requirements This could take a while 2022 08 18T07 31 26 353 INFO pip is looking at multiple versions of charset normalizer to determine which version is compatible with other requirements This could take a while 2022 08 18T07 32 26 952 INFO pip is looking at multiple versions of certifi to determine which version is compatible with other requirements This could take a while ```

average-flag-94768 · Answer

Cool glad you get repro without the private package

average-flag-94768 · Answer

Yeah from the terminal I run there is no feedback other than the time even when I set the level to trace

enough-analyst-54434 · Answer

So this looks like a case of just straight up slow backtracking to find a solution and the backtracking is hitting sdists it has to build along the way

enough-analyst-54434 · Answer

The Python ecosystem is perverse in that you often have to build an sdist to get metadata to continue the resolve walk

average-flag-94768 · Answer

Hmm I guess no easy solution no pun then

average-flag-94768 · Answer

It does resolve with poetry These are all based on poetry pyprojects after a couple of minutes but I guess that is a very unhelpful thing to say haha

enough-analyst-54434 · Answer

The solution is Pex upgrading its vendored Pip from 20 3 4 to 22 2 2 latest I ve been working hard on that and just tried in that branch with a wiped cache ```$ rm rf ~ pex amp amp time python mpex cli lock create output=lock json style=universal resolver version pip 2020 resolver target system linux target system mac indent=2 manylinux manylinux2014 interpreter constraint CPython==3 9 r ~ Downloads requirements txt real 1m2 519s user 0m54 418s sys 0m3 164s``` I m very close to done just adding tests and forking out PRs You can follow along here but I expect Pex 2 1 104 with the Pip upgrade will be released by this weekend and Pants mainline upgraded by Monday <https github com pantsbuild pex issues 1805>

average-flag-94768 · Answer

Thanks that s super helpful

enough-analyst-54434 · Answer

You re welcome I ll be happy to have Pex using the latest Pip again and going forward Pip has a good deal of problems but they also fix the important ones pretty promptly

average-flag-94768 · Answer

I m fairly new to python More of a Rust guy these days and I am really surprised by the complexity around having a monorepo setup that packages stuff into docker containers The fact that you have to create sdists to get metadata out is mindblowing to me So I am very thankful people like you are working on making it easier

enough-analyst-54434 · Answer

To be fair Rust and Go and these languages supported by big corps with alot of time money and hindsight have invested in shipping tools with the language up front Guido amp Co were just trying to ship the interpreter and all else was cobbled as time went by so 3rdparty handling is a jungle as a result Ditto Java although they had less excuse and had big $ available

average-flag-94768 · Answer

Oh yeah totally I m not criticising I m just surprised I ve had to setup too many c++ projects I know how hard these things could be just to get one or two deps working haha

enough-analyst-54434 · Answer

Yeah since Python has supported c extensions from day 1 and thus the variety of builds could be unbounded the wisdom was roughly let the user figure it out if they can don t straight jacket them so ship a system where you execute a setup py file Since that file is Python itself and not declarative you really have to execute it to get a dependency list since some of those in the wild actually have conditional logic

enough-analyst-54434 · Answer

And that was sometime in the early 2000s it just snowballed from there

enough-analyst-54434 · Answer

So Pip has a fairly uniquely hard problem to deal with Cargo Rust got it right from ~day 1

bored-glass-58755 · Answer

Reusing this thread to explain my requirement a bit more I want to generate stubs from proto files in my repo and then package and publish the generated files as a python library My questions are 1 What is the goal to generate stubs 2 Is it necessary to have the grpcio dependencies in 3rdparty python folder I have the dependencies listed in another directory in the same repo where I am going to be importing the generated stubs I am first building a conda environment which has those requirements fulfilled and then generating the stubs I am doing all of this in my local machine right now but would want to integrate this entire process in my CI once I get this working

hundreds-father-404 · Answer

Hey < bored glass 58755> happy to help although it would probably be better to have a dedicated thread because those questions don t quite relate to this thread about `generate lockfiles` hanging

bored-glass-58755 · Answer

Sorry I used the wrong thread by mistake thinking it was one of my other <https pantsbuild slack com archives C046T6T9U p1660763522826049|threads>

average-flag-94768 · Answer

I see the pex PR has gone in last week is there a timeline for when that change will be picked up in a pants release Looking at previous releases I imagine another week or two

enough-analyst-54434 · Answer

No timeline currently You caught most of us in the middle of a sprint on docker command runner support See the Pex release announcement for more <https pantsbuild slack com archives C18RRR4JK p1661713160748249 thread ts=1661713160 748249 amp cid=C18RRR4JK|https pantsbuild slack com archives C18RRR4JK p1661713160748249 thread ts=1661713160 748249 amp cid=C18RRR4JK>