Last thing (maybe) while I'm thinking on the tests. Tests that use

run_pants

would surely be over-saturating the system, no? You have the main Pants daemon process spawning N test subprocesses, where some subset of those run Pants (and I don't see us passing

rule_threads_core

or

process_execution_local_parallelism

) which then themselves allocate based on CPU count. Is that right?

absolutely I think we're getting resource contention. It's much faster to run a single IT at-a-time than run multiple at a time. And it helps explain the timeouts we get in CI

I'm wondering if we should refactor those to

rule_runner

where possible. Otherwise, limit the process count to like 1 or 2

Otherwise, limit the process count to like 1 or 2

That is probably excellent low hanging fruit

Everything being discussed sounds like good examples of bad creep that are good to fix - but isn't the motivating problem a large blowup at a point in time? Are you just trying to clear away cobwebs to make it easier to diagnose the big blowup or do I have that wrong?

I'm making note of cobwebs so that when we do fix the blowup we can clear those on the way out