The last piece of work really required to get off the ground with the Pants-compiles-its-engine would be incremental compilation. 🧠 🌧️ in 🧵

i don’t think that

sccache

is for incremental compilation: only for per-crate compilation

I can try it again, but my last test showed it did allow us to bypass re-compiling all of our dependencies in a sandbox

right: that’s a per-crate compilation cache, rather than a cache of incremental compilation (which is intra-crate)

Maybe "incremental" wasn't the right word, then? Are you saying that there'd be no benefit to

sccache

? Or that unlocking this "incremental" compilation would be better than using

sccache

?

example: when compiling the

engine

crate in

MODE=debug

for the first time, it takes ~2 mins on my machine: the second time it takes about 6 seconds

I'm not sure I completely follow 😅 It seems though like we'd want

sccache

for better 3rdparty and

incremental

for our own code?

right

OK, so either way seems like we need a named cache 🙂 Then the question is just if the incremental compilation cache is named-caches-safe 🙂

i know that they use file locking, but replies to this question suggest that some of the portions of the directory are not safe for sharing (the final outputs): https://stackoverflow.com/questions/58669482/is-it-okay-to-use-a-single-shared-directory-as-cargos-target-directory-for-all

But seems like concurrent runs for a single workspace is OK

yea

Although this seems like it "safe"? https://github.com/rust-lang/cargo/pull/2486 fromhttps://stackoverflow.com/questions/37471391/can-i-prevent-cargo-from-rebuilding-libraries-with-every-new-project

yea. i think that a project-specific directory would be a good defense.

Which is technically free if we just don't expose our little named cache to anything but this specific purpose 😛

which i think is definitely a useful feature, and also presents more subtlety than we might have imagined at first. the named cache setting would have to be an enum

Well, specifically I'm interested in a hack, so that we dont have to make these kinds of decisions now 🙂

I’ve been iterating on some rust code and quickly had like 50GB of artifacts in my target/debug directory (https://github.com/pantsbuild/pants/issues/9823). So, I’d be a bit concerned about a slightly-hidden truly append-only cache quickly expanding to consume the universe, and it being non-obvious how to resolve it, eg conventional rust tools/knowledge doesn’t apply (although conventional pants knowledge does) Thus, focusing on sccache for crate-level caching seems like a good path forward, because it can impose a max size on the cache and manage it itself. We can work out how to get cargo’s incremental compilation going in future (I hope we’ll keep

./cargo …

working outside the sandbox so we can keep the incrementality when building/testing rust directly)

@broad-processor-92400 : There is a ticket about reifying the

named_caches

a bit more... that same concern applies to ~everything in there.

There's also this, which I don't grok: https://docs.rs/cargo-hakari/latest/cargo_hakari/about/index.html

It generates a crate which gathers all things that have features, and then makes everyone depend on that. So just fixes the issue of different feature-flags on different

-p crate

invocations leading to unnecessary compilation times. Also, does this mean that proper rust support is in the works too? :o

Not from me, nope 🙂

Ah, hmm. So what does

Pants-compiles-its-engine

mean then? 😛 Just a more narrow support for that specific use-case?

Hah, right. A bit more narrow.

Not sure how

cargo

invokes

sccache

, but we maybe looking at a red herring:

The sccache command will spawn a server process if one is not already running,