https://pantsbuild.org/ logo
a

average-vr-56795

12/31/2022, 9:53 AM
Does anyone recognise this ?flaky test? https://github.com/pantsbuild/pants/actions/runs/3809714686/jobs/6482336850
Copy code
03:07:13.25 [ERROR] Completed: Run Pytest - src/python/pants/backend/python/lint/flake8/subsystem_test.py:tests - failed (exit code -6).
============================= test session starts ==============================
collecting ... collected 2 items

src/python/pants/backend/python/lint/flake8/subsystem_test.py::test_first_party_plugins PASSED [ 50%]
src/python/pants/backend/python/lint/flake8/subsystem_test.py::test_setup_lockfile PASSED [100%]

- generated xml file: /tmp/pants-sandbox-jpvlai/src.python.pants.backend.python.lint.flake8.subsystem_test.py.tests.xml -
============================== 2 passed in 14.23s ==============================

FATAL: exception not rethrown
Also seen on another test
Some googling suggests it's an issue around C++ exceptions and threading which insta-terminates a process
Not sure why it would start happening on https://github.com/pantsbuild/pants/pull/17907 though
But I've done ~5 reruns of CI and consistently hit the issue
Just tried reverting the recent rust update on the off chance - nothing else recent seems like it would obviously impact native code, but also, who knows what random python dependencies do
(Waiting on CI for that revert)
This is my biggest suspicion from the rust stuff: https://github.com/rust-lang/rust/pull/101077/
(But it's a stretch, I don't imagine anything is sending signals to our pytest processes...)
Wait, when I pushed a revert commit, CI decided to stop trying to run the python test shards?
squashes commits together so they trigger the superset of CI
Oh, no, I think it's just not triggering yet because the rust bootstrapping was slow 🙂
Also occurred after the revert, guess that wasn't it...
b

bitter-ability-32190

12/31/2022, 1:06 PM
I've been hitting it too 😔
c

curved-television-6568

12/31/2022, 1:19 PM
I’ve seen it once or twice as well, but not consistently.
and only on CI
a

average-vr-56795

12/31/2022, 1:45 PM
Cc @happy-kitchen-89482 who commented on the PR
h

happy-kitchen-89482

12/31/2022, 1:45 PM
👍
a

average-vr-56795

12/31/2022, 5:41 PM
Any pointers on how to grab a core dump out of GitHub Actions these days?
b

bitter-ability-32190

12/31/2022, 6:12 PM
None here other then upload files maybe 🤔
h

happy-kitchen-89482

12/31/2022, 8:55 PM
b

bitter-ability-32190

12/31/2022, 8:58 PM
I've only ever seen it on flake8 and pylint tests 🤔
h

happy-kitchen-89482

12/31/2022, 9:35 PM
That did resolve after a couple of retries
a

average-vr-56795

12/31/2022, 9:46 PM
I can happily remove my rust revert and keep retrying til we're green (🤞), but... Seems worth an investigation too...
b

bitter-ability-32190

12/31/2022, 9:58 PM
In recent history this has been from my changes 😥
a

average-vr-56795

12/31/2022, 9:58 PM
I'd pretty strongly assume that's an observer bias 🙂
b

bitter-ability-32190

12/31/2022, 10:11 PM
So one theory I have is this is coming from turning off coverage. It might not be related to coverage at all, but stems from time savings from coverage being turned off that creates a perfect storm. One of these issues was similar, where timing was important
It consistently being flake8/pylint tests is like our biggest clue though.
I also recently turned off the daemon when using run_pants. I feel colder about that one, but still not ruled out
a

average-vr-56795

01/01/2023, 1:37 AM
Well the release PR is merged at least :)
w

witty-crayon-22786

01/11/2023, 11:07 PM
my best guess is that this is related to https://github.com/pantsbuild/pants/issues/16105 … made a bit of progress toward resolving that one today with https://github.com/pantsbuild/pants/pull/17974