Had an interesting discovery at work today. We've had terrible CI flakiness on CI this week for ~2 weeks, which I think I finally triaged successfully today. The problems we've seen have been tests just randomly hanging on locks known to have been released, subprocess calls stalling, and other things that interact with the kernel. This is much worse under heavy load, so touching core files is a guaranteed flake pretty much (we use persistent CI nodes). As an example; I managed a repro today where an added test just doing

subprocess.Popen([sys.executable, '-m', 'our.app.py', '--help'])

took 14 seconds before the print. After digging around a bit, I noticed Pants fails to detect number of CPUs on our infra, and so we end up running with 16 cores instead of the 7 available. That's already 2x oversubscription, plus a ton of IO threads used by everything, also likely based on the 16 core count. The node otherwise looks very healthy, and the only metric that stands out in our dashboards is that during the duration of the test we add 250 PIDs, which goes down by 180 again after the test -- pantsd was a fresh start but I assume it kept running after. What's interesting is that we haven't really added more tests -- but our testing pexes are really heavy to build. That also correlates with the best reproduction case I have. The flaky test builds a pex fast but does a lot of IO and subprocesses. That then overlaps all our super-heavy PEX builds, and fails. The first heavy PEXs to build can also flake, but the last ones don't. Removing some tests or sharding immediately removes any flakes. As does having a warm cache, even with

--force

. I'm now curious whether something similar might be happening on the Pants repo. I've already checked that the CPU count detection works correctly there, which it does. But with only 2 cores available, the margins might be much smaller, and I imagine not all of the subprocesses Pants creates properly scale down to 2 cores -- it's a very low amount...

We currently set concurrency on the 2 core shards to 1, yet still hit issues

Yeah; but that process spawned can behave in any way, right? I'm wary of this with my Rust plugin right now, since Rust can go quite wide if it wants to, at least during early compilation. Even if I limited pants to 7 in my example, trying to compile 7 cargo targets would likely spawn 7 * 16 = 112 total rustc processes!

Oh definitely, it may be that our own tests cannot run reliably on 2 cores

I have a suspicion there may be issues that are papered over by those machines having persistent named caches too, e.g. https://github.com/pantsbuild/pants/issues/20108, and https://github.com/pantsbuild/pants/pull/19879#issuecomment-1741796438

Oh, that is interesting. Thanks!