https://pantsbuild.org/ logo
Join Slack
Powered by
Hi guys! Let's say I have a `Get(Process, ...)` th...
# plugins
s
Hi guys! Let's say I have a 
Get(Process, ...)
that runs a script that generates some binary and writes an absolute path to that binary in stdout. Then I want to use that binary to create 
RunRequest
and then use it in 
adhoc_tool
. There are two caveats though: 1) I can't put the produced binary into the Digest (for reasons) and 2) the generated binary might disappear and I might need to generate it again. I tried to solve these issues with 
ProcessCacheScope.PER_SESSION
and it works, but now when I run my 
adhoc_tool
that uses the generated binary it never caches the result and has to run every time even though I can guarantee that the generated binary is always the same. So the question is - is there a way to tell pants to build the binary if it doesn't exist, but stop pants from recalculating every downstream dependency of the binary?
g
Do you want 
PER_RESTART_SUCCESSFUL
instead, maybe? Would get you better caching while pantsd lives.
s
The thing is I want it to behave like 
ProcessCacheScope.ALWAYS
, for example, I generate the binary from some sources, and if I use this binary in a test I want the result of the test to be cached if the sources didn't change
and I want it to be cached in a cache service (which means persistently right?)
So basically I have the guarantee that the sources completely define the binary, and if the sources didn't change the produced binary will be identical between runs (idempotent)
this would be trivial if I could put it into the Digest, because pants would be able to manage all the caching, but I'm looking for some hook to tell pants to trust me
g
I don't quite understand the problem you're trying to solve, I think. For example, what is the output of the (remote) cache? That path will... more times than not be invalid, right?
I'm wondering if you can do a triplet build/validate/force-rebuild in some high level rule. But I'm not sure if that is going to work, as the primitives used to build those rules would still be subject to the same cache invalidation issues. Maybe there is an escape hook to be found in the EngineAware* group of types.
s
I don't quite understand the problem you're trying to solve
Here is what I'm up to https://github.com/pantsbuild/pants/pull/20873
g
OK; as a Nix noob... why can the actual binary not be cached?
s
because it can also be linked to some dynamically linked library, so you need to cache not only the binary, but everything it needs to run, and then you need to put it exactly into the same place so that it could find all the libraries
I could theoretically find all it needs to run, but that's just overkill since nix is already reproducible by design
so I basically have the source:
Copy code
nix_binary(
    name="dot",
    expr='(import (fetchTarball "<https://github.com/NixOS/nixpkgs/archive/06278c77b5d162e62df170fec307e83f1812d94b.tar.gz>") {}).graphviz',
    rel_path="bin/dot",
)
and it uniquely identifies the binary. If the source didn't change, then everything downstream could be cached even if the binary is not there yet (but you regenerate it)
g
Hmm. I'm wondering if something stupid like cache busting through the env would work here. Since the persistent part of the pants cache is only process-execution output, we know the rule will evaluate at least once per "pantsd" session. And the "result" from the rule would still hash the same if the output is the same.
I assume that nix is smart enough to not materialize the expr again if not needed? So a "refresh" is free and can be forcibly done once per pantsd run
s
yes, nix finishes almost instantly on the second run, but that's not the problem. The problem is that downstream targets get invalidated, like tests
g
Yeah, but if you bust the process invocation you don't need to set the cache scope I think, which would avoid that cascading invalidation (iirc)
s
I'm not sure I understand what you mean by "bust the process invocation"
g
So f.ex.
Copy code
env=FrozenDict(__HACK_CACHE_BUST=str(time.time()))
(The technique is very common in web workflows; https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching#cache_busting). Basically, in that situation you set very wide caching parameters but then override some part of the "request" when it has to be refreshed. Since we have an external cache here we can always cache bust and allow it to rerun nix.
s
env=FrozenDict(__HACK_CACHE_BUST=str(time.time()))
Do you mean 
await Get(ProcessResult, Process(..., env=FrozenDict(__HACK_CACHE_BUST=str(time.time()))
? I thought pants invalidates downstream targets cache if env changes, no? Then how does it know to rerun the rules when some other input changes?
2 Views
Open in Slack
PreviousNext
Powered by