Does anyone have any tips for tracking down cache misses?

To track down specific processes are not hitting cache or why a particular process didn't hit the cache?

Not sure about the difference here, but we're seeing cache misses on requirements pexes in CI a lot

well Pants runs a whole bunch of processes for each invocation, some may hit the cache and some may not.

We have BuildBuddy setup so I am expecting cross-run hits

generally cache misses are from the cache key changing which for a process is basically all of the cli arguments and environment plus the input root

is there a way to dump all this to see what the difference in cache key is?

Yes, you can make use of my OpenTelemetry plugin to dump work units to a jsonl file. Then you'll see all of the relevant values if you drill down on the metadata for process-related work units.

that's awesome, thank you!

You can find documentation on the process-related metadata here: https://github.com/shoalsoft/shoalsoft-pants-opentelemetry-plugin/blob/main/docs/streaming-workunit-handlers.md#process-metadata

When you say system-specific paths, you presumably mean ENV vars or something else?

yes, the paths might be in env vars or in CLI args

is there a way to say env vars shouldn't go into the cache key? we have some things like AWS_SECRET_ACCESS_KEY in our env vars...

not currently. Pants models process execution on REAPI and all environment variables are used as part of the cache key for that.

I feel like the better solution here is for pants to not be opinionated about the ENV var going into the cache key

The opinion came from using REAPI as the model and basically using REAPI protobufs for computing the cache key even for local cache.

I don't think there is any cost to deviating from REAPI's model for how cache keys should be constructed when users configure pants to act differently

I don't think there is any cost to deviating from REAPI's model for how cache keys should be constructed when users configure pants to act differently

I perceive the following costs though: 1. The REAPI cache assumptions are still in play for

remote_environment

so deviation creates a semantic difference between

local_environment

and

remote_environment

since Pants does not have control over REAPI servers. This semantic difference is a cost which would need to be documented and taught to Pants users. It is also a complication on anyone trying to contribute to Pants to learn how Pants handles processes. 2. One-off transaction cost for change: The REAPI assumptions are in place all over the Pants code base and so the change is very much non-trivial. Moreover, someone would actually have to do the work whether by contributing the work or hiring someone to do so. (From what I currently know, I don't believe any maintainers have an (inherent) interest in making this sort of fundamental change currently, but I could be wrong on that point. As for me, I do paid work on Pants; someone would need to hire me to design and make this sort of change.)

that's fair, I honestly didn't think pants worked with REAPI execution, I thought it only did caching

Frankly there are bunch of points on which I'd love to deviate from REAPI to try and innovate on how Pants does execution: 1. Environment variables / secrets management 2. Append-only named cache support (not currently supported with

remote_environment

except via a hacky advanced option and the need to control the server environment) 3. Persistent workers