boundless-ambulance-1116104/12/2023, 5:07 AM
? It seems I have 5 identical copies of torch, each of size 3.5G (
tells me the size of the
directory is 21G). When installing the same requirements in a virtual environment, the total size of the venv is 7G, with torch being 3.5G. This is after nuking the
directory and running
If I remove the 4 tests which transitively import torch, then nuke the cache and run
pants test ::
again, I only have one torch directory under
pants test ::
(but other packages are duplicated). So my current hypothesis is that pants creates a copy of torch (or other library) for every test importing them transitively, but I'm guessing it's a problem with my configuration.
enough-analyst-5443404/14/2023, 7:34 PM
boundless-ambulance-1116104/14/2023, 7:35 PM
task, caching the pants' named cache. I don't think I've had an OOM kill on my local build since I nuked the cache. When should those
be cleaned up normally?
enough-analyst-5443404/14/2023, 7:39 PM
invocation that creates them.
tool has no
boundless-ambulance-1116104/14/2023, 7:44 PM
be due to me modifying some code while
is running (which triggers the creation of the pex)? Because it seems to restart the process. (the OOM could explain the extra files in CI, but this could explain the extra files on my local build)
pants test ::
enough-analyst-5443404/14/2023, 7:46 PM
directly or indirectly when restarting then yes, but this does pretty much require a
from some source, OOM-Killer or otherwise.
processes racing to do the 1 time install of the wheel in that un-suffixed chroot dir. Once just 1 process wins, that version is never installed again and all racing processes cleanup their suffixed dirs before exiting. It's only if the racing processes are aborted that the cleanup goes un-done.
boundless-ambulance-1116104/14/2023, 7:55 PM
correspond to my 3rd party dependencies at different stages of the build process.
Do we really need to keep them all in the cache? After all, if I have the installed wheel in
2.1G /home/vsts/.cache/pants/named_caches/pex_root/installed_wheel_zips 2.2G /home/vsts/.cache/pants/named_caches/pex_root/downloads 2.2G /home/vsts/.cache/pants/named_caches/pex_root/pip_cache 7.9G /home/vsts/.cache/pants/named_caches/pex_root/installed_wheels
it doesn't seem I still need to keep the files in
? And if the installed wheels is invalidated because I need a new version of a library, that should also invalidate the corresponding directory in
. Can I safely remove some of those directories manually to decrease the size of my cache? Or should they normally be removed automatically, but were not because of the OOM error?
enough-analyst-5443404/14/2023, 8:02 PM
processes running while you issue the `rm`s. Taking 1 example though, downloads store sdists and those might be used to build wheels for new interpreters. So you know you use just 1 interpreter and the downloaded sdist can be chucked, but Pex doesn't know that. You might run it again in 5 minutes with
boundless-ambulance-1116104/14/2023, 8:04 PM