For my own understanding, it kinda seems like most...
# development
b
For my own understanding, it kinda seems like most Pants subprocess should be using
immutable_input_digests
as most won't be mutating inputs (save for
fmt
?) Is that true? Should we consider swapping the "default" so we have
input_digests
be immutable and
mutable_input_digest
for the alternative?
1
I suppose some tools aren't very symlink friendly. So there's them
f
"immutable" really means that the
immutable_input_digest
is something to be reused over multiple
Process
invocations
like SDKs etc. that are invariant over those
Process
invocations whereas the source files will change for each
Process
invocation
b
Oh hmmm, so immutable over time, not immutable in a sandbox?
f
right
you make a good point though about source files not changing in a sandbox
bazel creates symlinks from the sandbox back to the source tree in recognition of that
b
Yeah we have some large resources I'd love to "materialize" much faster 🙂
f
then might be an idea to think about for Pants but it would still be a separate idea from
immutable_input_digest
which is geared to things like unpacking the Go SDK once and then re-using it
b
Makes sense
f
(although re Go SDK, we used to merge it in as an input digest and switched to requiring it to be installed locally because immutable_input_digest support had not been implemented yet)
b
I also want this feature for
GeneratedSources
I have resource dependencies which are generated and are immutable over time (versioned sys libraries)
h
We indeed should likely be using immutable_input_digests in more places than we are now, e.g. https://github.com/pantsbuild/pants/issues/14070 Stu has only proposed that we should simplify the API before we port a bunch of stuff
1
b
I'm happy to be a sounding board/extra brain on the topic.
🧠 2
w
at one extreme, we could take the idea from https://github.com/pantsbuild/pants/issues/14070 (includelist paths that should be immutable) and invert it to excludelisting paths which should be mutable
👀 1
w
Not that it would speed anything up, but is there already the ability to create a sandbox with
ro
properties on files?
w
but the current implementation would leave a lot of garbage behind… there is some amount of a hint around re-use inherent in marking something
immutable
currently, because we keep it on disk for the lifetime of
pantsd
@wide-midnight-78598: somewhat: immutable_inputs are marked readonly, since they will be shared between processes.
w
I just started thinking about 3rd party deps tools being good citizens in the sandbox, and basically enforcing that with attrs, and maybe even users
Kinda tangential to this convo though - so I'll just mindlessly ponder
w
mmm, so! there is another potential change here as of late. the
brfs
filesystem hosts digests as a FUSE filesystem. we discussed using it as an alternative to implementing something like immutable inputs, but at the time, FUSE was far too unstable on macOS to be a solution. in the last week or so though, https://www.fuse-t.org/ was announced, which replaces the unstable portion on macOS, and apparently improves performance
2
oh. but i had forgotten about this comment: https://github.com/pantsbuild/pants/issues/12716#issuecomment-982131890 … that’s probably still true? i.e., we would basically want to implement the
immutable_inputs
in terms of FUSE, rather than necessarily having the entire sandbox be FUSE.