I think I've found a deadlock related to https://github.com/pantsbuild/pants/issues/16969. I'm looking at a completely hung pantsd which, after running

pants test <a path>

just fails to exit. Specifically it seems to happen after running a fairly large number of processes, I guess because higher lock contention increases risk of it happening. I've done an inventory of the threads; and they break down as follows:

T1: python code
T2..T5: complete_workunit -> stderr_use_color
T6: python code
T7: complete_workunit -> stderr_use_color
T8: log_from_python -> stderr_use_color
T9..T10: complete_workunit -> stderr_use_color
T11..T14: notify-rs/fs-watcher
T15: parked tokio
T16: python code
T17: maybe_set_panic_handler -> stderr_use_color
T18: maybe_display_render -> ProgressBar::state

The only Python code of note is T6 which originates in

src/nodes.rs

, so likely Rust calling Python. The other two only has Python code on them. All three end up in

PyThread_acquire_lock_timed

either way, while most native threads are trying to take the destination lock to check if we can use color. I can't actually find what's holding the lock to the destination, which is confusing.