broad-processor-92400
10/20/2022, 1:55 AM~/.cache/pants/named_caches
. Based on https://www.pantsbuild.org/docs/using-pants-in-ci#directories-to-cache, there's suggestion that it should cache tools/PEXs. However, if I run a CI build with a full cache hit (key = hash of all lock files) on named_caches
and an empty lmdb_store
, I still see a lot of lines like the following (plus corresponding Completed
):
For lint/check (ends up being ~20s of wall time spent building the pexes):
Starting: Building isort.pex from isort_default.lock
...
Starting: Building extra_type_stubs.pex
Starting: Building mypy.pex from 3rdparty/python/mypy.lock
Starting: Building 59 requirements for requirements.pex from the 3rdparty/python/default.lock resolve: ...
Starting: Building requirements_venv.pex
For testing (ends up being 2.5 minutes of wall time building pexes):
Starting: Building 42 requirements for requirements.pex from the 3rdparty/python/default.lock resolve: ...
...
Starting: Building pytest_runner.pex
...
I'd be expecting that the ~2GB in named_caches
would eliminate the 3 minutes of building all these pexes, even if nothing else is cached, but that's apparently not the behaviour. Is there something I may be doing wrong? (A few more details in the thread)named_caches
too (i.e. fresh named_caches
and lmdb_store
)
• Cache key:
pants-named-${{ runner.os }}-${{ hashFiles('pants.toml') }}-${{ hashFiles('3rdparty/**/*.lock') }}
• The output above is from commands like ./pants lint check ::
and ./pants test ::
.
• The output from [stats].log = true
includes info like (#
comments mine):
Counters:
...
local_cache_requests_cached: 0 # due to empty lmdb_store, presumably
...
local_store_read_blob_size
...
sum: 14329918098 # 14GB suggests that _some_ cache is being used, but a build with empty named_caches has a similar number
• pants version: 2.13.0witty-crayon-22786
10/20/2022, 3:20 AMbroad-processor-92400
10/20/2022, 3:24 AMnamed_caches
at all, because the time spent restoring/saving it seems to be more than it savesStarting: Building ... Completed: Building ...
lines are cached in lmdb_store
(or in a remote cache, if using), rather than named_caches
, even the tool ones (using default lockfiles) like black/isort?witty-crayon-22786
10/20/2022, 9:41 PMlmdb_store
(local) or remote cachecold-sugar-54376
10/24/2022, 7:50 PMnamed_caches
and only caching the lmdb_store
?broad-processor-92400
10/24/2022, 9:18 PMwitty-crayon-22786
10/24/2022, 10:12 PMcold-sugar-54376
10/25/2022, 9:03 PMbroad-processor-92400
10/25/2022, 9:17 PMpants-lmdb-${{ runner.os }}-${{ hashFiles('pants.toml') }}-${{ hashFiles('**/BUILD') }}-${{ hashFiles('subdir/**/*') }}
and then we restore from truncated versions of that (we're only using pants in subdir
at the moment). We're hoping to switch to a remote cache soon-ish though.