Hello, is there an easy way to delete or limit the...
# general
m
Hello, is there an easy way to delete or limit the pants cache? It regularly fills up to 20GB or more, and surely not all of this is needed. Or do I need to manually delete it from time to time?
w
For now, manual deletion. There has been some investigation from time to time to call down to the underlying tools in order to prune them (and then have Pants prune it’s own stuff).
There are some demos/examples of that floating around, but nothing yet in the repo
If you do the following, where is the bulk of your storage coming from
du -h -d2
m
Copy code
% du -h -dZ *
 12G	lmdb_store
7.1G	named_caches
228M	pants_dev_deps
b
@many-vase-25409 FWIW Pex put the effort in to deal with its portion of the
named_caches
entry. There is now a parallel-safe
pex3 cache prune
that does the right thing by default and prunes all caches entries older than 2 weeks for Pex. Pants people have not adopted this though, despite driving it with these issues: + https://github.com/pantsbuild/pants/issues/11167#issuecomment-2453762939 + https://github.com/pantsbuild/pants/issues/18294#issuecomment-2625180595
👍 1
m
Copy code
(venv) jan@tbe named_caches % du -sh *
1.4G	coursier
 37M	mypy_cache
1.5G	pex_root
  0B	python_build_standalone
So the
pex_root
cache grows because with every code change a new pex needs to be build and the old ones are not pruned? The
coursier
cache is probably fairly stable if dependency don't change
w
https://github.com/pantsbuild/pants/issues/11167 This has been around a bit - I have an internal cache "management" plugin (super dumb one) that I can try to turn into the first part of this. As I just suggested in the associated ticket, instead of sitting on this any longer to solve time-based or dep-based pruning, I think it makes more sense to just get an info + purge option, and defer pruning to a later PR. It works for NPM 🤷 https://docs.npmjs.com/cli/v9/commands/npm-cache?v=true
b
Is it safe for NPM? 2 repos using NPM via Pants on the same machine (so two pantsds), 1 purges while the other is, say, running tests.
w
Sorry, I wrote that poorly. I intended to mean "not having a prune option seems to work for NPM" - not the act of purging yet
b
Still applies to purge.
It was nontrivial to add this support to Pex anyhow.
I basically see no way to do it without reader/writer file locks generally
w
As I said, my private cache management plugin is pretty dumb and just does a handful of checks with some locking before purging, but as it only runs on my machine and CI - the checks aren't extensive enough to put into general availability. The actual Pants one that gets built out would likely defer to each tool, and then Pants would try to manage the stuff it creates otherwise
b
Ok. Well it'll never happen.
w
Ok.