#18507 Investigate use of `/tmp` / `tmpfs` / persistent storage for execution roots Issue created by stuhood #18153 currently proposes switching from hardlinking to symlinking of large input files, because in some cases

/tmp

is mounted on a different filesystem as a

tmpfs

(check

df -h $(mktemp)

on the relevant machine). But from a quick survey of developers, mounting

/tmp

as a

tmpfs

has become more rare (recent Ubuntu and macOS mount it on a traditional filesystem). Too small a

tmpfs

can also result in

Os { code: 28, kind: StorageFull, message: "No space left on device" }

if enough concurrent processes are running. Benchmarks of using for `--local-execution-root-dir`: •

tmpfs

• a persistent directory •

/tmp

- the current default (on a machine with it mounted persistently) ... show that using a

tmpfs

is ~25% faster than using a persistent directory, and 45% faster than using

/tmp

(for an unknown reason). This is before the actual optimization suggested by #18153 is implemented though (which will reduce the actual IO that results from creating large files, by using hardlinks/symlinks in more cases). * * * Based on this, I'm proposing that we: 1. default

--local-execution-root-dir

to the same filesystem as the store 2. add explicit GC for

--local-execution-root-dir

• likely by nesting it under its pid, and periodically deleting orphaned directories (rather than relying on tmp cleanup) 3. continue to use hardlinking in #18153 • with detection/fallback to either copying or symlinking (TBD after benchmarking, since copies would avoid visibly changing rule sandbox behavior) if

--local-execution-root-dir

has been set to a different filesystem This will allow for a good default on machines that don't have an available

tmpfs

(which seems common), while still allowing folks to move the

--local-execution-root-dir

if they have explicitly created a large enough

tmpfs

and would like to use it. pantsbuild/pants