Hi! So we use the remote cache sidecar to host the cache, which is backed by S3. In S3, we set a retention period to limit how big that cache gets. Is there a local equivalent? One of our devs just reported having to delete their 260GB pants cache on their M1 notebook….

Nope.

Kk, wanted to check if nothing automated. It’s not that surprising for us, since pants is heavily part of our dev cycle, and theres a bunch of docker going on.

Does Pants cache docker stuff? I thought we just defer to docker - which has its own caching?

Yea you’re right - i’m not too certain then why it’s so large - but our collection of 3rdparty deps is large, so I suppose maybe that?

You can

tree

inside and see what's up. That number just seems way too wildly large. As you update deps and stuff, that will increment, as well as more caching, but you're like still 10x higher than I've seen before 😄

well our dev blew up their cache so I’ll have to take a look at my own

Some (older) discussion at https://github.com/pantsbuild/pants/issues/11167

There's another Slack discussion somewhere. One of the issues comes down to how well (or not well) the various tools that cache can handle partially missing files. So, let's say that

pex

can do a pretty good job incrementally finding and re-downloading files, but other tools might not be able to

fwiw, we started using pants a couple of weeks ago and my

~/.cache/pants/

is 8GB. the project has a lot of heavy dependencies for video editing and ml tools, so i'm not surprised it's big but I wonder if thats from us doing something 'wrong' for it to be so big so quick

I'd expect the size to be very use case dependent. (Torch!)

$ du -hsc ~/.cache/pants
42G	/home/ecsb/.cache/pants
42G	total

Gotta love Torch 🙃

I’m surprised my own is <10GB. Digging a bit into some of the larger packages (i’m giving you side-eye, torch):

$ ls -1 ./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949
torch-1.13.1-cp39-none-macosx_11_0_arm64.whl
torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.5abd1931a475453d95c1275be068a5f0.work
torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.6efc7ab82c774a7f9af763315c6b0a02.work
torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.aa1acbc3cb8c48bbb027cb251a7a9325.work
torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.fd369f0187ce46b1b4fc26b361ec1b2a.work

And they are identical…

$ du -h -d1 ./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949
245M	./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949/torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.6efc7ab82c774a7f9af763315c6b0a02.work
245M	./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949/torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.aa1acbc3cb8c48bbb027cb251a7a9325.work
245M	./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949/torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.5abd1931a475453d95c1275be068a5f0.work
245M	./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949/torch-1.13.1-cp39-none-macosx_11_0_arm64.whl.fd369f0187ce46b1b4fc26b361ec1b2a.work
245M	./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949/torch-1.13.1-cp39-none-macosx_11_0_arm64.whl
1.2G	./named_caches/pex_root/installed_wheels/e0df902a7c7dd6c795698532ee5970ce898672625635d885eade9976e5a04949

Any guidance to what could cause this?

Feels like pex atomic directories working in parallel? But I'm kinda guessing (and assuming that the contents are, in fact, identical)

A sample of my local cache:

5.4G	~/.cache/pants/lmdb_store
5.3G	~/.cache/pants/lmdb_store/immutable/files
5.3G	~/.cache/pants/lmdb_store/immutable
2.9G	~/.cache/pants/named_caches
2.8G	~/.cache/pants/named_caches/pex_root
1.7G	~/.cache/pants/named_caches/pex_root/installed_wheels
521M	~/.cache/pants/named_caches/pex_root/pip
514M	~/.cache/pants/named_caches/pex_root/pip/24.0
511M	~/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache
511M	~/.cache/pants/named_caches/pex_root/downloads
510M	~/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache/http-v2
289M	~/.cache/pants/lmdb_store/immutable/files/ba
261M	~/.cache/pants/named_caches/pex_root/installed_wheels/53e1c33c6896583cdb9a583693e22e99266444c4a43392dddc562640d39e542b/torch-2.0.0-cp310-none-macosx_11_0_arm64.whl
261M	~/.cache/pants/named_caches/pex_root/installed_wheels/53e1c33c6896583cdb9a583693e22e99266444c4a43392dddc562640d39e542b
258M	~/.cache/pants/named_caches/pex_root/installed_wheels/53e1c33c6896583cdb9a583693e22e99266444c4a43392dddc562640d39e542b/torch-2.0.0-cp310-none-macosx_11_0_arm64.whl/torch
230M	~/.cache/pants/lmdb_store/immutable/files/d2
199M	~/.cache/pants/lmdb_store/immutable/files/e7
199M	~/.cache/pants/lmdb_store/immutable/files/8d
174M	~/.cache/pants/lmdb_store/immutable/files/d0
174M	~/.cache/pants/lmdb_store/immutable/files/c9
174M	~/.cache/pants/lmdb_store/immutable/files/11
171M	~/.cache/pants/named_caches/pex_root/installed_wheels/be91c6c81e839613c6f3b15755bf71789839289d0e3440fab093e0708516ffcf/opencv_contrib_python_headless-4.10.0.84-cp37-abi3-macosx_11_0_arm64.whl/cv2
171M	~/.cache/pants/named_caches/pex_root/installed_wheels/be91c6c81e839613c6f3b15755bf71789839289d0e3440fab093e0708516ffcf/opencv_contrib_python_headless-4.10.0.84-cp37-abi3-macosx_11_0_arm64.whl