Is there a plan to provide a maximum cache size Mine got to

Question

Is there a plan to provide a maximum cache size Mine got to 30GB and don t seem to shrink neutral face

enough-analyst-54434 · Answer

Yes You can follow along here <https github com pantsbuild pants issues 11167>

witty-crayon-22786 · Answer

`~ cache pants lmdb store` should already be garbage collected by `pantsd` periodically it s only `~ cache pants named caches` that aren t collected currently

witty-crayon-22786 · Answer

which of those has the bulk of the usage

enough-analyst-54434 · Answer

I thought we gc d but did not compact the LMDB

enough-analyst-54434 · Answer

Or something along those lines

witty-crayon-22786 · Answer

that s correct but it runs often enough that it should cap the total usage at a high water mark

melodic-thailand-99227 · Answer

I have like 20 venvs each a copy of the entire dependecy tree

melodic-thailand-99227 · Answer

image png

enough-analyst-54434 · Answer

The venvs are created with hardlinks So for example on my machine 2 5 actual GB ```$ du sh ~ cache pants named caches pex root venvs 2 5G home jsirois cache pants named caches pex root venvs ``` But apparent 28GB if you count each hard link as new data l ```$ du lsh ~ cache pants named caches pex root venvs 28G home jsirois cache pants named caches pex root venvs ``` I m unfamiliar with ncdu Do you know what it s counting Is it hard link aware

polite-garden-50641 · Answer

somewhat of a know issue <https pantsbuild slack com archives C046T6T9U p1636626099143000>

polite-garden-50641 · Answer

enough-analyst-54434 · Answer

I looked into this and found a win for named caches Although symlinks and hardlinks provide roughly the same savings obviously more inodes for hardlinks The hardlinking was not sharing ` pyc` amongst venvs Switching to symlinks solves this and for example for toolchain ```$ du sh ~ cache pants lmdb store tc venv symlinks ~ cache pants named caches pex root tc venv symlinks 179M home jsirois cache pants lmdb store tc venv symlinks 1 1G home jsirois cache pants named caches pex root tc venv symlinks $ du sh ~ cache pants lmdb store tc venv hardlinks ~ cache pants named caches pex root tc venv hardlinks 179M home jsirois cache pants lmdb store tc venv hardlinks 3 4G home jsirois cache pants named caches pex root tc venv hardlinks``` I ll be getting out a PR for this shortly

witty-crayon-22786 · Answer

Very cool I guess it makes sense with so many fewer inodes but still not obvious

witty-crayon-22786 · Answer

Probably a lot faster to create too

enough-analyst-54434 · Answer

Its ~nothing to do with the fewer inodes It s to do with pyc files now being all shared

enough-analyst-54434 · Answer

The pyc files are only created later wen the venvs are executed against by whatever intepreter Now all the work of creating a say Python 3 6 pyc for module foo is shared as is the space Before with hard links neither the effort nor space were shared Each venv had its own unique pyc

enough-analyst-54434 · Answer

Similar to the same silent problem we suffer with loose sources in Pants execution chroots that are only adjoined via PEX EXTRA SYS PATH we always re compile these and never share There there is no real space savings since the chroots are ephemeral but we waste effort repeatedly Benjyt had noted this when working getting django repo working with Pants some tie ago O 1sec time spent re compiling pycs for every IT

enough-analyst-54434 · Answer