curved-manchester-66006
08/31/2023, 5:59 PMIf you're not using a fine-grained remote caching service, then you may also want to preserve the local Pants cache at $HOME/.cache/pants/lmdb_store.Does that mean with a REAPI cache there will not be a
lmdb_store
directory, but pex_root
and friends will all still be under named_caches
? Under what conditions would you want to use a REAPI cache and cache all of lmdb_store
? (the "may" from the advice)curved-television-6568
08/31/2023, 6:02 PMbroad-processor-92400
09/01/2023, 12:22 AMlmdb_store
cache when using a remote cache, the remote cache is a second cache that can be used to fill the local cache when something isn't there... but if something has already been run locally or already downloaded from a remote cache, pants will be able to find/reuse the result from that local cache.
Using a remote cache will be better than coarse-grained "tar a directory and upload it"-style caching of lmdb_store
(e.g. what github actions does with its fully transient runners), because each job will only download what it needs... whereas lots of stuff in lmdb_store
will become out of date and just be uselessly uploaded/downloaded.
That is, if you're using a CI service like that, and using a remote cache, caching lmdb_store
is likely wasted time.
However, if you're using a CI service with persistent runners (i.e. separate jobs run on the same "physical" machine), preserving lmdb_store
between runs is likely helpful, even with a remote cache: if something has been downloaded from the remote cache by one job, it'll be saved locally and won't need to be re-downloaded for a future job. But, this is just ensuring that directory isn't cleared out.
That said, the "may" there is technically different: it's suggesting that if you're not using a remote cache, there's times when you might not want to preserve lmdb_store
. I guess you might find doing fresh builds every time is faster than coarse grained caching of that whole large directory.
So, in summary, the four options:
• use remote cache, preserve `lmdb_store`: for persistent CI runners, where lmdb_store
can be saved with no effort/cost
• use remote cache, no preserve `lmdb_store`: transient CI runners, where lmdb_store
would need to be archived and uploaded in bulk
• no use remote cache, preserve `lmdb_store`: when a remote cache isn't possible (e.g. cost or time), and saving lmdb_store
makes builds faster/cheaper
• no use remote cache, no preserve lmdb_store
: when a remote cache isn't possible (e.g. cost or time), and saving lmdb_store
makes builds slower/more expensivecurved-manchester-66006
09/05/2023, 1:28 PM