Before I start digging a deep hole for myself; has anyone attempted to keep file

mtime

intact in file ops in Pants? It's going to be necessary at some point for the Rust backend to scale properly. The caching on content works great until any file changes, at which point Cargo rebuilds everything because the sandbox has fresh mtimes.

I think the caching protocol doesn’t store them (and, indeed, storing them is suboptimal, and would cause lots of misses), so… my two-second thought is that sounds like a deep hole! Only half joking: I wonder if it would be easier to make cargo’s cache content based. (I personally haven’t tried, to answer your actual question)

The idea would be to store but ignore them for digests etc. So I don't think cache misses would increase, we'd just be restoring more data when we read from the cache.

Hm, I believe the remote CAS/byte store computes hashes directly on the whole serialised protobuf that is stored, and verifies on download (some servers may verify on upload too?). I don’t know how the local one works.

Ah, I hadn't even considered remote stores. If those APIs would even be able to contain the data in a safe way.

Just brainstorming some other options that you may've considered: • invoke rustc directly (I guess this is reimplementing the build half of cargo...), which is, I believe, what bazel does, and a lot of work... • continue invoking cargo, but hack in something with sccache, i.e. cargo still does all the rustc invocations, but a lot of of them are fast, when files don't change... although Josh found some issues that made this harder to use https://pantsbuild.slack.com/archives/C0D7TNJHL/p1695308830998599?thread_ts=1695233195.030429&cid=C0D7TNJHL

Thanks! Yeah, reimplementing cargo is definitely out of scope. Sccache is more interesting. Since we could install sccache into a named cache we could maybe work around those issues. I also had a look at "fixing" cargo and it looked surprisingly easy. So we'll see... Just need to find enough time to do something.

Oh, yeah, time. That'd be handy for me too. Lemme know when you find a good source. Even wilder idea: starlark build rules support for Pants, so Pants can reuse more of bazel's rust support.

See https://github.com/rust-lang/cargo/issues/6529 for a bunch of history of the cargo stuff. They're interested in accepting the feature, and I don't think it's even that complicated, but would need breaking up into reasonably reviewable chunks

If the status quo changes re

mtime

, I want to know, because I've been working on packaging via nFPM, and

mtime

handling is a big chunk of code (I have to prevent nFPM from using any

mtime

in the process sandbox by giving nFPM a list of every file saying what the mtime should be.)

I swear I saw ed page had an issue somewhere talking about it as well

I put a note about this in my

mtime

field help text (emphasis on the part I think most relevant to this conversation):

Though nFPM supports pulling mtime from the src file or directory in most cases, the pants nfpm backend does not support this. Reading the mtime from the filesystem is problematic because Pants does not track the mtime of files and does not propagate any file mtime into the sandboxes. Reasons for this include: git does not track mtime, timestamps like mtime cause many issues for reproducible packaging builds, and reproducible builds are required for pants to provide its fine-grained caches.

See also: https://reproducible-builds.org/docs/timestamps/

I agree to a point with that, but I think it's confused about cause and effect. All files have mtimes, including every single file in the sandbox already. It's just a shitty "now!!" mtime that means it has no meaning for build caching. Any tool that we wrap in a sandbox only sees that timestamp, so if it tries to use it meaningfully it's never going to be reproducible.

WOuldn't setting timestamp (even epoch 0) be slower than doing ~nothing?

Yes. 🙂 But doing nothing means it's "now" from the FS. So all files always change.

OK, my vote would be that we don't change the default. As defaulting to "fastest" is the best experience, since NN% of processes don't care. Then, users can opt-in to slower-but-possible-better-caching-from-the-process.

Maybe a new set of rules can take a standard digest of sources, inspect the workspace sources, and return some kind of annotated digest that includes the mtime? So a rule has to opt into using the heavier-weight digest (prob a subclass of the standard digest objects), and the mtime details will only be captured if requested.

Yeah, at least the things that depend on changing files.

my only experience building rust comes from using pants in the pants repo where it has to compile itself first. 😛