curved-manchester-66006
08/31/2023, 5:59 PMIf you're not using a fine-grained remote caching service, then you may also want to preserve the local Pants cache at $HOME/.cache/pants/lmdb_store.Does that mean with a REAPI cache there will not be a
lmdb_store directory, but pex_root and friends will all still be under named_caches? Under what conditions would you want to use a REAPI cache and cache all of lmdb_store? (the "may" from the advice)curved-television-6568
08/31/2023, 6:02 PMbroad-processor-92400
09/01/2023, 12:22 AMlmdb_store cache when using a remote cache, the remote cache is a second cache that can be used to fill the local cache when something isn't there... but if something has already been run locally or already downloaded from a remote cache, pants will be able to find/reuse the result from that local cache.
Using a remote cache will be better than coarse-grained "tar a directory and upload it"-style caching of lmdb_store (e.g. what github actions does with its fully transient runners), because each job will only download what it needs... whereas lots of stuff in lmdb_store will become out of date and just be uselessly uploaded/downloaded.
That is, if you're using a CI service like that, and using a remote cache, caching lmdb_store is likely wasted time.
However, if you're using a CI service with persistent runners (i.e. separate jobs run on the same "physical" machine), preserving lmdb_store between runs is likely helpful, even with a remote cache: if something has been downloaded from the remote cache by one job, it'll be saved locally and won't need to be re-downloaded for a future job. But, this is just ensuring that directory isn't cleared out.
That said, the "may" there is technically different: it's suggesting that if you're not using a remote cache, there's times when you might not want to preserve lmdb_store. I guess you might find doing fresh builds every time is faster than coarse grained caching of that whole large directory.
So, in summary, the four options:
• use remote cache, preserve `lmdb_store`: for persistent CI runners, where lmdb_store can be saved with no effort/cost
• use remote cache, no preserve `lmdb_store`: transient CI runners, where lmdb_store would need to be archived and uploaded in bulk
• no use remote cache, preserve `lmdb_store`: when a remote cache isn't possible (e.g. cost or time), and saving lmdb_store makes builds faster/cheaper
• no use remote cache, no preserve lmdb_store : when a remote cache isn't possible (e.g. cost or time), and saving lmdb_store makes builds slower/more expensivecurved-manchester-66006
09/05/2023, 1:28 PM