Hey everyone, we have a CI setup on our pants monorepo with self-hosted runners. We have been using multiple runners, and each one of those uses a persistent

named_caches

directory, which is separated by sha256 of

default.lock

This works well within a single runner setup, where a job would always be rescheduled on the same runner every time, which can reuse the existing named_cache from previous. However in multiple runner setup, we have no guarantee, that the same runner will be used on subsequent runs, which will contain the named_caches built from previous run. I have thinking of using AWS EFS to mount named_caches directory which would be shared by all the runners. Has anyone tried this kind of setup? Appreciate your thoughts and ideas on this

Theoretically Pants' only uses

named_caches

in a concurrency-safe way, so it may work to have many runners reading/writing to the same cache (assuming EFS itself handles concurrent changes appropriately), but also, it's definitely possible that there's bugs in either the tools Pants invokes or in the way that Pants invokes them, that may mean that it's not 100% concurrency-safe and thus you may end up with "weird" behaviour occasionally that may be fixed by clearing the caches. I've not personally tried anything like this, though.

We tried the shared local cache approach (as well as remote cache, that work wonderfully) in our CI and yep seeing weirdness. Things like

22:06:07.88 [INFO] Completed: Building pytest_runner.pex
22:06:07.99 [ERROR] 1 Exception encountered:
Engine traceback:
  in `test` goal
ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:
stderr:
[Errno 2] No such file or directory: '/ci_cache/pants/named_caches/pex_root/venvs/3adf9388b7ac52b3bb22fd4b5b829292bef89a33/f0cba893d66b1c9fbeabd90148143eda391645ff.lck.work/lib/python3.8/site-packages/six.py'
Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

and

File "/ci_cache/pants/named_caches/pex_root/unzipped_pexes/46d58d3bb664a3b74c3bb3cbdd126f27ca55c81b/.bootstrap/pex/venv/pex.py", line 78, in _copytree
    os.link(src_entry, dst_entry)
OSError: [Errno 31] Too many links: '/ci_cache/pants/named_caches/pex_root/unzipped_pexes/46d58d3bb664a3b74c3bb3cbdd126f27ca55c81b/__pex__/__init__.py' -> '/ci_cache/pants/named_caches/pex_root/venvs/46d58d3bb664a3b74c3bb3cbdd126f27ca55c81b/5740eb338971daae690ec6e90ccfd02a26ca1263.lck.work/lib/python3.8/site-packages/__pex__/__init__.py'
Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.