has anyone figured out a fix for this issue beside restarting your machine? I’ve run into it a few times in the last two days:

Exception: Could not initialize store for process cache: "Error making env for store at \"/private/var/folders/cf/z3ktw6dn467gvm24fgx8ft6c0000gp/T/tmpk1bj57ql/processes/7\": No space left on device"

yes @hundreds-breakfast-49010 has been working on a fix, but is blocked by us having no idea what’s causing this one specific CI failure. https://github.com/pantsbuild/pants/pull/8892 We’d love your eyes on it if you have some time

thanks for the context. I’ll take a look if I have some cycles this week.

Yeah I would love to merge this commit but I am not sure what's making CI not pass with this

as daniel pointed out on that PR, it doesn't seem like a fix... the bug appears to be around not cleaning things up in tmp

@witty-crayon-22786 yeah it's a mitigation, not a fix

yea, true. but that is only if we segfault

Nothing abnormal happened to me while testing between now and the last time I fixed this by restarting

@witty-crayon-22786 I don't think it's just segfaults, it's any time execution crosses FFI into python between when a rust object is created and when the program exits

that... doesn't seem likely.

hm, this reminds me that in my own replications ofthe running out of disk space thing, I saw an error that looked like it was coming from pthreads based on googling

so, maybe what's happening is that if python throws an exception in the main thread (which is what happens in hte out of disk space case), a separate thread doing process execution stuff gets aborted

python is not producing that exception: rust is

I don't recall seeing that error message when I was testing locally on my system

sure

I wonder if maybe what you're seeing on your machine and what I'm seeing on my machine are different

possibly. That error was coming from python when it was trying to init the scheduler

oh wait, that's the error @early-needle-54791 was seeing

from what I can tell. python is initializing the engine, which fails, and then python raises that exception, which comes from an error that has been cast into a python exception type, or maybe just a string handle?

so, the "out of space" error is independent from what causes the out of space error: especially since it requires a restart to resolve.

on linux I'm able to recover from the error by clearing out a bunch of flies in /tmp

agreed. Not sure how to do the equivalent on mac

FWIW this is something I’ve come across regularly as of late, typically hitting the error (out of space) after 3-5 runs of

./pants --no-v1 --v2 test

that are only resolved after a reboot. If there is any further digging / context I can provide to help diagnose this (for macs - I’m on a MBA with the latest macOS release) please let me know.

and the space is all taken up inside

/tmp

?

it’s using

/private/var/folders/fv/<...>

on mac, errors look like

Exception: Could not initialize store for process cache: "Error making env for store at \"/private/var/folders/fv/9ly0y0dj1g946plb36nm4s1m0000gn/T/tmpv8o2rfya/processes/8\": No space left on device"

Checking that dir after the run fails shows the referenced dirs not there

ok. yea, i repro after a few runs of roughly that command. but there is no change in

tmp

size before and after.

Which is what I was seeing also

which makes me suspect virtual memory rather than actual files.

if this fixes the problem, do we still need to figure out the problem with https://github.com/pantsbuild/pants/pull/8892 ?

Imo, no... Unless we think that the

whl

size improvement is worth it

#8933 looks to fix the issue for me! i’ve done ~10 test runs so far without issue