https://pantsbuild.org/ logo
#general
Title
# general
c

cold-soccer-63228

07/28/2022, 6:48 PM
To what extent is the performance impact of caching / not caching
.cache/pants/named_caches
? I noticed that this directory is extremely large (on the order of gigabytes), and caching it is somewhat expensive because it takes a long time to download (and upload) this cache
On the other hand,
.cache/pants/setup
and
.cache/pants/lmdb_store
are pretty small and fast download + upload
b

bitter-ability-32190

07/28/2022, 6:50 PM
Thats the "cheat" cache, where we point tools to if they are multi-process safe. Namely what you're likely seeing is a large
pex_root
, which among other things, is where all your downloaded dependencies are stored 🙂
Feel free to run
du
on the subdirs and report back worst offenders
c

cold-soccer-63228

07/28/2022, 6:50 PM
I see… so the reason for this cache being really big is due to the size of my dependencies?
b

bitter-ability-32190

07/28/2022, 6:51 PM
Don't take my word for it,
du
the subdirs 😛
If
pex_root
is large, then yes 🙂
c

cold-soccer-63228

07/28/2022, 7:55 PM
Ahh, turned out to be an issue with me constantly uploading to the cache and not nuking when it got too big…
Thanks for the pointers Josh!
b

bitter-ability-32190

07/28/2022, 8:01 PM
@witty-crayon-22786 any pointers for trimming the PEX root? Seems like it'll just grow and grow and grow?
w

witty-crayon-22786

07/28/2022, 8:02 PM
https://www.pantsbuild.org/docs/using-pants-in-ci discusses this and recommends a particular script
c

cold-soccer-63228

07/28/2022, 8:02 PM
Yeah, the nuke script 🙂 I didn’t have that previously but will probably add it in
Thanks all!
w

witty-crayon-22786

07/28/2022, 8:02 PM
but https://github.com/pantsbuild/pants/issues/14364 discusses how the advice there could be more precise: the entire
pex_root
is not useful to cache, per-se
h

hundreds-father-404

07/29/2022, 4:28 PM
Also check out this blog about how remote caching makes this much much better https://blog.pantsbuild.org/better-ci-cd-caching-with-pants/
2 Views