pantsbuild #development

Channels

welcome

github-notifications

development

hundreds-breakfast-49010

12/30/2019, 10:41 PM

so not all that big compared to multiple copies 159MB and other things use it

hundreds-breakfast-49010

12/30/2019, 10:42 PM

and I'm definitely seeing most of the space in /tmp being taken up by process-execution-* directories

hundreds-father-404

12/30/2019, 10:42 PM

That sounds plausible that cleanup is not working properly. Note that to fix this issue, you must restart your machine. It doesn’t matter if the

./pants

invoke already finished

hundreds-breakfast-49010

12/30/2019, 10:43 PM

also, there are other directories in /tmp that look like

/tmp9ud9ds6g

, that appear to have nothign except a single copy of native_engine.so

😶 1

hundreds-breakfast-49010

12/30/2019, 10:43 PM

not sure where those are coming from

hundreds-breakfast-49010

12/30/2019, 10:44 PM

@hundreds-father-404 I didn't have to restart my machine to fix this, I just deleted the process-execution-* and other tmp* files out of /tmp

hundreds-breakfast-49010

12/30/2019, 10:44 PM

again I wonder if this is a linux-mac difference

hundreds-breakfast-49010

12/30/2019, 10:45 PM

we're using the rust

tempfile

crate in the rust engine, maybe this crate specifically does something different on a mac vs linux?

hundreds-breakfast-49010

12/30/2019, 10:48 PM

@aloof-angle-91616 I think the files aren't being deleted if disk space happens to completely fill up while there are still pending EPRs running. in that case, pants hits the out of disk space error, crashes, and then fails to clean up the tempdirs that caused the problem to begin with

👍 1

hundreds-breakfast-49010

12/30/2019, 10:48 PM

but if it manages to avoid ever completely filling up disk space, then they'll get deleted like any other temp directory

hundreds-breakfast-49010

12/30/2019, 10:49 PM

so maybe one thing we could do is handle runing out of disk space more gracefully

aloof-angle-91616

12/30/2019, 10:49 PM

yes, but the error doesn’t go away when i

rm -rf /private/var/folders

hundreds-breakfast-49010

12/30/2019, 10:50 PM

maybe that's not the place on a mac where these particular tempfiles are being written? or maybe the next run of pants just happens to also fill up disk space, even if you cleared the remnants from the first run

hundreds-breakfast-49010

12/30/2019, 10:51 PM

anyway, 1) we should probably handle running out of disk space more gracefully 2) we should figure out if we can make native_engine.so smaller

aloof-angle-91616

12/30/2019, 10:51 PM

i think that the machine just literally waits until restart to garbage collect

hundreds-breakfast-49010

12/30/2019, 10:51 PM

or figure out how to not materialize it many times

aloof-angle-91616

12/30/2019, 10:51 PM

i can try to investigate this

hundreds-breakfast-49010

12/30/2019, 10:52 PM

yeah I want to reiterate that I am really only familiar with how tempfiles work on linux, not on mac os

aloof-angle-91616

12/30/2019, 10:52 PM

i don’t understand why the space isn’t registering as deleted yet and that seems like the right problem to solve rather than adding complexity to avoid materializing files

hundreds-breakfast-49010

12/30/2019, 10:53 PM

so my understanding of what the rust

tempfile

crate is actually doing, is using file system APIs to create a randomly-named directory in

/tmp

. then when the handle to that directory goes out of scope in Rust, it runs a

drop

method that deletes that same directory

hundreds-breakfast-49010

12/30/2019, 10:54 PM

but that only works if the program is running long enough to drop that handle and run

drop

. if the program crashes before that, it will just never delete the tempdir it created

👍 1

hundreds-breakfast-49010

12/30/2019, 10:54 PM

in linux everything in

/tmp

goes away on a system reboot, and I assume the same is true on a mac

👍 1

❌ 1

aloof-angle-91616

12/30/2019, 10:55 PM

yes, which is what we’ve been seeing i think (that restart makes it go away)

aloof-angle-91616

12/30/2019, 10:59 PM

i can research this but am about to go on a flight

hundreds-breakfast-49010

12/30/2019, 11:01 PM

@hundreds-father-404 mentioned that there was a bash script that made sure that native_engine.so is available to get packaged into the Digest that gets materialized to the pytest EPRs - I take it that's build_support/bin/native/bootstrap_code.sh ?

aloof-angle-91616

12/30/2019, 11:14 PM

it might be an optimization though when spawning many, many processes on a local machine to have a set of “canonical” materialized files (for commonly-materialized files) and then use symlinks in /tmp to those canonical files

hundreds-breakfast-49010

12/31/2019, 12:26 AM

I've been looking to see if there's anything we can do to make sure that pants at least cleans up its tempfiles, even if it exits in these circumstances

hundreds-breakfast-49010

12/31/2019, 12:27 AM

https://github.com/pantsbuild/pants/blob/master/src/python/pants/engine/scheduler.py#L468 <- is it just me, or does that

return PANTS_FAILED_EXIT_CODE

never get run? because

_trace_on_error

always throws an exception that nothing else catches?

adamant-mouse-92628

12/31/2019, 12:28 AM

@hundreds-breakfast-49010 one idea is to use mdmfs (or tmpfs). We could then clean up ourself each time we start

adamant-mouse-92628

12/31/2019, 12:29 AM

we'd still clean up the FS itself at completion