there is a `pantsd` crash on the Linux-ARM64 shard...
# development
there is a
crash on the Linux-ARM64 shards on
, but i am completely out of steam for investigating it today. maybe Monday.
… as one last diagnostic effort, i am going to restart a bunch of Linux-ARM64 shards on
that were killed by other issues that Benjy and i investigated, in hopes that that will allow for identifying when this issue began.
goddamn computers
I poked at this a bit - running
./pants --pantsd-max-memory-usage=100MiB
passes in an SSH session, while
fails with the same error as the CI shard.
So possibly the oom killer is overreacting
I haven't been able to track down its logs yet
But according to the pants logs, pantsd is receiving SIGTERM, but I believe the oomkiller sends SIGKILL
So maybe it's something else
Hmm, and sometimes it does pass without lowering
So I don't know what is going on
Going to bed now
will poke some more tomorrow
Hmm no, the SIGTERM was unrelated (from killing pantsd manually I guess), there is nothing useful in the pants log.
The sigbus hypothesis was from an exception file in .pants.d on the worker.
Now in a terminal on the box things seem to run OK (last night they were more consistently failing), but this gets logged:
[WARN] Executor shutdown took unexpectedly long: tasks were likely leaked!
Things still consistently fail in CI though
Yeah, seeing some "Bus error" in the exceptions from last night
So that is a strong hypothesis
The machine obviously has plenty of available RAM, so it's not that
And we've rebooted, right?
Going to try nuking ~/.cache/pants for the gha user and reboot again 🤷‍♂️
heh, that seems to have worked
At least for now
thanks a lot for looking at that.