fwiw: i have a strong suspicion that the reason fo...
# development
w
fwiw: i have a strong suspicion that the reason folks have reported seeing “Downloading” tasks for things like PEX more often than you’d otherwise expect is that the node ends up blocked waiting for an IO slot to check whether we actually need to download
👍 2
👍🏻 1
my hope is that https://github.com/pantsbuild/pants/issues/13112 will improve that situation
💯 1
f
That... would make sense; I saw this enough at times where I started watching network activity related to it and never saw any actual downloads
w
yea. it’s been an ongoing struggle to decide “how much” of any particular node to render. there are various points at which you can block and wait, and many of them should count as the task running (time spent actually downloading)… while others shouldn’t (time spent waiting to check the cache to see whether you should download).
even trickier because we don’t have actual queues on IO… just a limited number of IO threads vs
@rule
threads
f
I thought tokio just parked threads on IO waits so a pool could be shared among various thread-like things (fibers?)
w
yea, tokio tasks will be parked. but the “queue” implicitly represented by being parked is internal to tokio’s scheduler… would need to know internal details to figure out “how many tasks are parked waiting to run as spawn_blocking” essentially
for process executions we do apply our own bound on concurrent execution (via an async Semaphore), and so we have some visibility into the fact that we’re going to be waiting
f
pretends to understand what's going on in a real rust program
w
in the end, it’s the curse of async, regardless of language 😉
👻 1