we had this discussion before about "materialize" and "write" in this channel

(i don’t feel strongly about it)

Okay so it was this channel. I was trying to search for it in TC slack and couldn’t find it

(same)

I prefer

write

. Asher pointed out it’s helpful for people who don’t speak English as their first language. And it fits with using simpler terms for things like

AddPrefix

and

RemovePrefix

But willing to use another word, of course

simpler english is better english most of the time imho

i got my hackles raised about "provenance" because for some reason i didn't remember ever learning that word before for example so by precedent at least but idk

simpler is good as long as non-ambiguous. but yea, shrug here*

I feel better about it because the only place this now shows up is

workspace.write_digest()

. There’s no longer a top-level type such as

WriteDigest

happy to do

write_digest_to_build_root

as well. I think I prefer that?

nah,

workspace.write_digest

is great

hm. i finally see why people aggressively re-export

@rules

when they think that they are going to need them.

Thoughts on renaming

SnapshotSubset

to

DigestSubset

? In https://www.pantsbuild.org/v2.0/docs/rules-api-file-system, almost everything uses

Digest

and I’m presenting it as “The core building block is a Digest”

Snapshot

only augments

Digest

with information possibly useful to you.

yes!

oh I asked this april 29th lol

Cool. Then, the only reason I didn’t proceed is that the intrinsic was written as returning

Snapshot

, instead of digest. I’m rewriting the 3 intrinsics that currently do that. The engine will automatically upcast those digests to snapshots if the rule author wants

if we ever find ourselves waiting on the

Digest -> Snapshot

operation too long, it is possible to cache that specific action persistently with an extra ShardedLmdb, so i wouldn't worry about the performance implications of this at all

i think using a digest as the base here will lead to greater correctness

I suspect this may speed up perf because we often only need a

Digest

at these call sites. We were only requesting

Snapshot

because the engine forced it

Excellent. You can now do

await Get(DigestContents, PathGlobs)

.

fwiw: computing a Digest requires a Snapshot, so not sure it is necessarily cheaper

i had roughly looked at a hierarchy like this for the “compute Paths without Snapshotting them” change:

Snapshot    <- Digest <- Paths <- PathGlobs
DigestContents <-

Okay. Either way, the Python rules are much more ergonomic with this change. The Rust path can be optimized for performance still

Is this paragraph true?

To set environment variables, use the parameter

env: Mapping[str, str]

. You generally should not read from

os.environ

because this reduces reproducibility. Instead, hardcode the value or use a [

Subsystem

option](doc:rules-api-subsystems).

this is a general and low-priority (for the 2.0 launch) observation, but it seems to me like it would be nice if we could consolidate the various

lift_

methods defined rather haphazardly in

<http://nodes.rs|nodes.rs>

“Error code 26: Text file busy” 👀 https://travis-ci.com/github/pantsbuild/pants/jobs/366361027#L528 This is on John’s PATH PR, where we add a

./script.sh

file via

CreateDigest

to be able to locate any arbitrary binary on the

PATH

. It looks like “Text file busy” is when multiple processes are trying to modify the same file. I don’t know how that could happen with the engine, where each

Process

is its own tmpdir?

Fun, I think that Pants has been running system Pex this whole time for me? John’s PR made me realize that. See https://travis-ci.com/github/pantsbuild/pants/jobs/366361029#L505. We’re trying to run

pex

, rather than something like

./pex

.

how does the native_engine.so get built in remoting? Is it built locally and then uploaded to the CAS, or is it built in the remoting environment?

the

Engine traceback

currently includes only rules with human readable names. should it include all of them?

i have a PR from a while ago (https://github.com/pantsbuild/pants/pull/8706) that would allow you to try/catch an

await Get(...)

and also provides a python-level traceback with exact line numbers — it does this by raising the exception from the Get() generator instead of early exiting from the engine, and preserving the stack for the traceback. the PR also keeps the engine traceback, which is displayed below the python-level traceback. would the ability to have a python-level traceback exception with line numbers/etc be useful? if so, i can clean up that PR.