#development
Title
h
happy-kitchen-89482
02/08/2023, 5:01 PMUgh, I'm modifying the
init-pantspants --versionIt should take 45 seconds, but it'll hang until the job gets canceled.
e
enough-analyst-54434
02/08/2023, 5:22 PMSo the recent bug fix added thread pool shutdown explicitness. IIRC it used a 250ms timeout but maybe a place to double check.
Also, should that action be running --no-pantsd? Ideally we'd be able to use pantsd in CI without reservation but I have not tracked the status of that.
h
happy-kitchen-89482
02/08/2023, 6:12 PMI figure it should use pantsd or not based on the configuration in pants.ci.toml? I wouldn't want to hard-code either way.
But yeah, the thread pool shutdown seems a possible culprit
w
witty-crayon-22786
02/08/2023, 6:34 PMbut with a warning if it actually took that long: https://github.com/pantsbuild/pants/blob/d12d1b9408c84c5ee5a20841d31d7100b578fb62/src/rust/engine/task_executor/src/lib.rs#L255-L261
if you have an example repro PR though, i can take a look
h
happy-kitchen-89482
02/08/2023, 6:43 PMIt repros ~50% of the time in that job
I will SSH in and see if I can repro manually
w
witty-crayon-22786
02/08/2023, 6:53 PMk. back in about 45 minutes to take a look.
h
happy-kitchen-89482
02/08/2023, 6:55 PMNaturally I can't repro via SSH
Notably, this only happens on linux, not macos
e
enough-analyst-54434
02/08/2023, 7:26 PMIt does seem to be the case that Pants concurrency bugs seem to always surface on Linux and I imagine, by inference, that Linux kernel locking / thread scheduling must be finer grained than Mac / BSD allowing more diversity in orderings to happen more regularly.
h
happy-kitchen-89482
02/08/2023, 7:54 PMDue to the pattern of when this was happening vs not I thought it might be related to named-caches being retrieved vs not retrieved from the CI cache, but I can confirm this is not the case.
Going for a run, will dig more later
Can also confirm that sometimes rerunning the job in the exact same state makes it work and sometimes not. So this does seem like a race condition.
FWIW I cannot (at least so far) reproduce this with pantsd turned off
w
witty-crayon-22786
02/08/2023, 8:07 PMblargh. well… i’m also planning on disappearing for a few hours. i’ll take a look this evening, but anything you can do to narrow the repro would be really helpful
e
enough-analyst-54434
02/08/2023, 10:20 PMI just got 2 nice big hangs in CI immediately after upgrading a scie-pants IT to use 2.15.0rc4 to work around `FATAL: exception not rethrown`:
Linux: https://github.com/pantsbuild/scie-pants/actions/runs/4128694155/jobs/7133454406
Mac: https://github.com/pantsbuild/scie-pants/actions/runs/4128694155/jobs/7133454550
Not helpful except that this time a Mac too and another case of
--pantsdI have never seen hangs before, just the variants of `FATAL: exception not rethrown`; so my money is on the 2.15.0rc4 fix breaking other things here.
w
witty-crayon-22786
02/08/2023, 10:48 PMA backtrace from one of those would be worth its weight in gold.
I also noticed the other day that the SSH GitHub action can also be triggered only for canceled steps
e
enough-analyst-54434
02/08/2023, 10:56 PMThat is definitely not true from my own past personal sshing
But I guess I don't know what you mean. Its its own step; so you need to get to that ssh step. Could be cancel, green or a never fail on a prior step to ignore its failure. AFAICT its ~anything that gets you to that magic ssh step allows ssh'ing.
w
witty-crayon-22786
02/08/2023, 11:05 PMYeah, I didn't try the cancel one first hand: I do know that the failure hook works. If a previous step fails it will trigger the SSH handler
e
enough-analyst-54434
02/08/2023, 11:08 PMThere has got to be a better way to do this. Debugging concurrency issues via CI ssh is pretty crappy. Maybe I can try some local hobbling exercises this weekend to see if I could have debugged this stuff locally.
h
happy-kitchen-89482
02/08/2023, 11:11 PMYeah, plus I haven't been able to reproduce any of this in SSH
w
witty-crayon-22786
02/08/2023, 11:22 PMYea. If the cancel handler works, then it would allow for catching them when they happen naturally in CI. I'd love a better harness for local testing... if this was pure rust code, we could use loom for at least some cases.
But 9/10 times it's the GIL + some other lock.
e
enough-analyst-54434
02/08/2023, 11:23 PMIt probably needs 2 days of code reading vs a debugger. Sometimes just understanding is way better. Right now we lean on you too much for that.
If you have to catch with a tool, it probably means you'll just get trapped again in my book, since you don't really fully grok it.
Ok, got a repro hang and was able to attach gdb and get full thread logs. I'm having a look now and will post shortly.
And I added a crappy analysis. I have no idea really what's going on with flows here. My head does not yet contain the model. I'll step back from this investigation. Let me know if you need anything more.
w
witty-crayon-22786
02/09/2023, 7:01 PM2 Views