so it’s only accessible from

@goal_rule

s

and yea: putting things in

dist

, but it’s also capable of writing files anywhere else in the working copy, so it’s used by the

fmt

goal to re-format code, and

tailor

to write update BUILD files

So is the idea that accessing "hermetic" outputs is strictly done via

output_directories

and

output_files

?

This leads me to another question: is there a notion of just implicitly capturing all writes to the

Process

chroot? (I'm assuming that all processes are run in some sort of chroot-like temp dir)

Like, setting aside whether it's a good idea, does it make sense to say

output_directories=(".",)

?

it only allows for literals

and that just propagated from bazel’s API

but since output directories are supported, if you want to capture “a bunch of unknown stuff”, writing to (or moving into) an output directory and then capturing that directory will generally work

… i don’t know off the top of my head whether

output_directories=(".",)

would work, but i think that it would rarely be what you want, because it would re-capture all of the inputs as well

Thanks!

And even those it seems need to be treated with care, as though they’re ephemeral to a 

Process

 invocation but might give you performance improvements because of sideband caching

yea, exactly. they’re only for well behaved append-only caches.

ok, made some doc edits: added

--no-*-cleanup

to the

Process

page. was already on the debugging page: https://www.pantsbuild.org/docs/rules-api-tips#debugging-look-inside-the-chroot

for example, (1) set the cache directory location to a known value in the input root that won’t conflict with other usages,

__coursier__/cache

and (2) then capture specific jar resolutions from the cache as output directories, e.g.

__coursier__/cache/https/repo1.maven.org/maven2/io/circe/circe-generic_2.13/0.12.3/

(or as an output file if that works better). then you’ll have a digest for each resolved jar and could construct a specific input root from those digests

(tangentially, I’ve been thinking about “synthetic” process executions which could help with splitting a single batch process execution into multiple individual “synthetic” process executions. think for example about turning the single result from

coursier fetch foo:1.2.3 bar:5.6.7

into two synthetic actions for

coursiser fetch foo:1.2.3

and

coursier fetch bar:5.6.7

)

(given that, in some sense, the execution model prefers creating command-lines for process executions that have already been run and thus are cached)

yea, it looks like

fetch

is transitive, so you can’t split it

I'm curious though, how smart is the underlying CAKVS about deduplicating identical files within a given digest? I had assumed that was going on under the hood, so I shouldn't worry too much right off the bat about attempting to manually split up individual jars within a

Digest

I think that the primary reason not to do that though would be if you wanted independent cache keys for each download... particularly for coursier, the named_caches should make that less of an issue. BUT the named_caches are an extension to the remote execution protocol which we have not implemented in any servers yet.

Even if the resolve/fetch were split, you'd still be able to take advantage of the named caches for each of them independently (The named cache discussion is over here: https://groups.google.com/g/remote-execution-apis/c/mA87-IDNpec/m/Nt9GL1NrBAAJ)

Although, re hermetic/non-hermetic: unless each split-out fetch was URL+expected digest, it still might not be reproducible. Nonetheless.