Is there any mechanism to benefit from durable rem...
# plugins
a
Is there any mechanism to benefit from durable remote caching other than through
execute_process()
? Are there any plans to expose such a mechanism to plugin writers? E.g., a new decorator, or a config argument for
@rule
?
We have a use case where some huge, arbitrary collection of build/`execute_process()` sub-steps (across multiple plugins) could all together be considered one overarching build step, and we have enough information at time-zero to fully feed a caching decision. Imagine a very large source tree broken into many, many small sub-trees. Each sub-tree takes many steps to build, but we can know that if (e.g.) the digest of the subtree, etc, is the same as the last time we built this sub-tree, then the end result (after a great many intermediate substeps) will be identical. We benefit from all of the many intermediate
execute_process()
invocations yielding cached results, but non-trivial build-/plugin-level work still needlessly happens, notably dynamic-dependency resolution (which is deterministic if a large-enough circle is drawn). I suspect a carefully-constructed
execute_process('pants <...>')
would be a working, horrific kludge to accomplish what I'm imagining, but that entails significant overhead in multiple dimensions. I can imagine such capability would be a big hammer requiring care to not violate invariants, but in our case the fault would lay with the plugin author.
h
Interesting. One possibility, at least conceptually, is to cache the state of the rule graph (this is currently memoized but not cached even locally). So the invocation of high-level rule that triggers all those processes is itself cached
Currently there are a couple of special cases of local caching that don't involve a
Process
(e.g., dep inference results), but they are very bespoke.
But I have long wondered about having some sort of "fact caching" that is a level above processes. As you say, this would require the plugin author to be very, very careful that everything that needs to be mixed in to the cache key actually is
Again, rule invocations have well-defined cache keys, so maybe that's a thing.
c
durable remote caching other than through
For durable remote caching one would be somewhat constariend by the REAPI protocol, right?. My -- at a distance -- understanding is that people have done creative things on the content-addressable-store corners of the api but on the Action side things need to be stretched to look like a Command.
f
Isn't a Command just a string? Using whatever mechanism makes it work with remote_execute set to false and remote_cache set to true, you should be able to shove an entry in the cache with any random SHA as the command string, no? (He says not having read the code, or the protos)
c
Yeah I think it's possible in principle. Prior related art: https://github.com/buildbarn/bonanza I was trying to tease apart "cache more stuff in lmdb" from "cache more stuff remotely"
a
Thank you! Context is helpful.