#17282 Include/exclude and heuristic-based `immutable_inputs` New issue created by stuhood

Process.immutable_inputs

is currently configured as a mapping from a file path to a

Digest

to materialize at that path. It is merged into the

input_digest

of a

Process

to form a "complete"

Digest

of inputs. But this separation is a bit unfortunate, because it means that using

immutable_inputs

(which should be fairly widely used: see discussion on #14070) means a lot of change to how your

Process

is constructed. * * * From an API-changes perspective, we should: 1. remove the guarantee that inputs are mutable by default, to allow us to automatically choose the strategy that we use for each path, and symlink wherever we choose to. • Then, by default, we should automatically use symlinking for files or directories over a certain size, etc. 2. convert the existing

immutable_inputs

list into a flat list of paths which should be _forced_/includelisted to be materialized as immutable (regardless of the heuristic from above), but which must already exist in the

input_digest

(maybe "force" should go in the name?

force_immutable_paths=

...?). • This will mean that unlike today (where you need to change how your

Process

is constructed to stop adding an input to the

input_digests

and instead add it to the

immutable_inputs

as a dict), you would only need to add a path to the

immutable_inputs

, without changing your

input_digest

. 3. add an

mutable_inputs

(or

force_mutable_paths=

...?) flat list of paths to act as an excludelist that prevents a path from being materialized as a symlink. * * * From an implementation perspective, this would look like moving the support for creating the symlinks for

ImmutableInputs

from an explicit step before running a process: pants/src/rust/engine/process_execution/src/local.rs Lines 641 to 662 in </pantsbuild/pants/commit/08410e5b94d3e3aa86b2538ee1991d964e27e72c|08410e5> ...to a thing that happens inside

fn materialize_directory

when an includelisted path is encountered: pants/src/rust/engine/fs/store/src/lib.rs Lines 1188 to 1201 in </pantsbuild/pants/commit/08410e5b94d3e3aa86b2538ee1991d964e27e72c|08410e5>

materialize_directory(_helper)

would take either: 1. a reference to a collection of paths which were includelisted as immutable 2. a trait or function which implemented the heuristic for which `directory::Entry`s / paths to includelist ...and a reference to

ImmutableInputs

, and then would create a symlink to the

ImmutableInputs

whenever it encountered one of the includelisted paths. pantsbuild/pants

#17282 Include/exclude and heuristic-based `immutable_inputs` Issue closed as completed by thejcannon pantsbuild/pants