#development
Title
e
enough-analyst-54434
12/12/2022, 9:38 PMI have a (2x so far) reproducible case of https://github.com/pantsbuild/pants/issues/15312 in scie-pants CI for Pants 2.16.0.dev1+git298409b3 (298409b3a4d7914c29a3bea5098b55698967f658). This is not using --no-pantsd which some issue work that closed this issue out as dup seems to focus on.
Seen here:
+ https://github.com/pantsbuild/scie-pants/actions/runs/3678739650/jobs/6222363684#step:7:464
+ https://github.com/pantsbuild/scie-pants/actions/runs/3679867403/jobs/6224812319#step:8:463
Looks like:
Copy code
...
>> Verifying PANTS_SHA is respected
Bootstrapping Pants 2.16.0.dev1+git298409b3 using cpython 3.9.15
Installing pantsbuild.pants==2.16.0.dev1+git298409b3 into a virtual environment at /home/runner/.cache/nce/638e71475c6feb8228292d052407a5cf6b7813caf3630e80d42711d1081e8c1d/bindings/venvs/2.16.0.dev1+git298409b3
New virtual environment successfully created at /home/runner/.cache/nce/638e71475c6feb8228292d052407a5cf6b7813caf3630e80d42711d1081e8c1d/bindings/venvs/2.16.0.dev1+git298409b3.
21:30:47.58 [INFO] Initializing scheduler...
21:30:48.17 [INFO] Scheduler initialized.
2.16.0.dev1+git298409b3
Fatal Python error: PyGILState_Release: thread state 0x7f156c0010d0 must be current when releasing
Python runtime state: finalizing (tstate=0x14a06a0)
Thread 0x00007f1585b0ac00 (most recent call first):
<no Python frame>
Error: Command "dist/scie-pants-linux-x86_64" "--no-verify-config" "-V" failed with exit code: None
Error: Process completed with exit code 1.I'm not sure this is that useful ... I do not hit the issue on my Linux box, only GH actions Ubuntu 20.04 CI runs so far.
b
bitter-ability-32190
12/12/2022, 9:40 PMI feel like I ran into something similar a while back. I'll respond when I get time if I find anyhting useful
e
enough-analyst-54434
12/12/2022, 9:40 PMAnd its not exercising the run goal either, just
./pants -VYeah, @bitter-ability-32190 I was referring to your close of that issue as dup and then further work Stu did.
Your issue didn't look exactly like #15312 - but very close. Mine appears to present identically to #15312.
I'm probably going to get lazy and just pick any other older sha than 298409b3a4d7914c29a3bea5098b55698967f658 to test PANTS_SHA with, but it appears badness with blocking is back / whac-a-mole continues. I'll record this all somewhere on the Stu issues.
w
witty-crayon-22786
12/12/2022, 9:55 PMis
pantsdbut … i believe that i have also seen this on macOS for the PEX building command from the release process (
./build-support/bin/release.sh build-universal-pexe
enough-analyst-54434
12/12/2022, 10:11 PMIt is during teardown from the looks of the output I included above.
./pants -VSo this is nice in minimal, although only remotely reproducible so far.
And I've only seen it 2x and only on Ubuntu 20.04 in CI and only on that PANTS_SHA version. 2.14.0, 2.12.1 and 1.30.5rc1 are all stable.
w
witty-crayon-22786
12/12/2022, 10:59 PMIt is during teardown from the looks of the output I included above.withand it prints out the version successfully and then the CPython thread state error../pants -V
pantsd./pants -Vpantsde
enough-analyst-54434
12/12/2022, 11:02 PMYeah, should be the client.
I just run a series of -V against different Pants versions
w
witty-crayon-22786
12/12/2022, 11:03 PMweird. but yea, that would explain why it was in the foreground.
e
enough-analyst-54434
12/12/2022, 11:03 PMThis is #3 of 4 serial runs.
You're on the review that has the test harness.
w
witty-crayon-22786
12/12/2022, 11:05 PMi have no idea what the client might be doing asynchronously… but it would be a new class of race condition than is described in https://github.com/pantsbuild/pants/issues/16105, since the client shouldn’t actually have an Executor to worry about, afaik.
…but… switching to the native client would definitely resolve it. no more Python on the client =P
e
enough-analyst-54434
12/12/2022, 11:06 PMBoo
Dodge
(e)VADE
w
witty-crayon-22786
12/12/2022, 11:06 PM“Multiple Birds with One Stone”
is how i like to think about it 😃
or “Cutting the Gordian Knot” as Benjy referred to it recently re:
sciebut … i believe that i have also seen this on macOS for the PEX building command from the release process (this was unrelated, (un)fortunately. but is fixed here at least: https://github.com/pantsbuild/pants/pull/17785), in a similar spot../build-support/bin/release.sh build-universal-pex
e
enough-analyst-54434
12/12/2022, 11:21 PMYeah. But it's an ongoing issue in not the client. We all have 0 time. I'll try to dig deeper here this week.
w
witty-crayon-22786
12/12/2022, 11:34 PMBut it’s an ongoing issue in not the client.what do you mean by “not the client”?
e
enough-analyst-54434
12/12/2022, 11:36 PMThis is still whac-a-mole over in pantsd right? That's why you have a ticket.
Its a general problem with no clean solution / guard rails yet IIUC at a high level.
w
witty-crayon-22786
12/12/2022, 11:38 PMi don’t think it’s `pantsd`: i thought that we concluded above that it was probably the client. e.g. https://pantsbuild.slack.com/archives/C0D7TNJHL/p1670886334122209?thread_ts=1670881112.587059&cid=C0D7TNJHL and https://pantsbuild.slack.com/archives/C0D7TNJHL/p1670886158750069?thread_ts=1670881112.587059&cid=C0D7TNJHL
in any case: will review the scie stuff tonight, and can look at anything you file here.
e
enough-analyst-54434
12/12/2022, 11:40 PMI agree it's not Pantsd, I thought it was an instance of a problem we do have in pantsd which your ticket tracks.
IOW, we can punt the client as you suggest since it will die some day, but pantsd will not so investment in me tracking down what why how for this should be non-throwaway effort.
w
witty-crayon-22786
12/12/2022, 11:42 PMbut pantsd will not so investment in me tracking down what why how for this should be non-throwaway effortmaybe. i think that the
pantsdbut yea, i don’t know what we don’t know.
e
enough-analyst-54434
12/12/2022, 11:43 PMOk, well, any mystery in code is bad. So I will try to track this down.
👍 1
I personally get the horrors with stuff like this until I understand fully.