this is very noticeable… so noticeable, that it doesn’t seem like a boiled frog situation. it seems to have changed a significant amount since i last looked at it.
did you make any more progress on the bisect?
a complete shot in the dark would be that due to the relative difference between
and debug mode, that this had something to do with Rust upgrades
01/17/2023, 6:27 PM
Maybe a silly question, but does everything fit comfortably in RAM on the runners? And do they have swap disabled? We had an issue where pylint would slurp up all our RAM and thrash (or oom when we disabled swap), I wonder if something similar could be happening?
01/17/2023, 6:32 PM
yea, it should. there was a bunch of triage done in the linked thread narrowing this down to graph solving in debug mode
01/17/2023, 6:58 PM
I will try the bisect again, on a machine that doesn't arbitrarily slow down its cores for power management reasons
And automate it
01/17/2023, 6:59 PM
you might start by doing a before/after on the last few rust upgrades, or i can