The setup: -

remote_cache_read/write = true

in

pants.ci.toml

-

<grpcs://cache.depot.dev>

as REAPI endpoint - Linux x86_64 CI runners, pantsd enabled -

[stats].log = true

for diagnostics What we see: - Run A:

remote_cache_requests: 374, cached: 0, write_successes: 374

- Run B (empty commit, same branch, same code):

remote_cache_requests: 374, cached: 0, write_successes: 374

- Same

CODEARTIFACT_AUTH_TOKEN

value between runs (verified) - Lint and Package goals on the same runs DO show increasing cache hits What we verified locally: - Running the same test twice on macOS with

-ldebug

, the

Run Pytest

Process struct is byte-for-byte identical (same argv, env, input_digest) - Local remote cache round-trip works -- we see accumulating hits across runs - The 3

PerRestart*

processes are correctly excluded from remote cache (as expected per PR #16920) - Cache scope on test processes is

Successful

What we ruled out: - Rotating credentials (stable shared token, verified identical between runs) - Platform mismatch (both runs on same Linux x86_64 runner type) - Depot-side issues (1 GB stored, 14-day retention, no limits) Question: What else in the Process cache key could differ between two CI runs on the same branch with the same code? We're wondering if

concurrency_available

,

PATH

, or some other runner-specific field could vary between invocations. Is there a way to dump/compare the actual REAPI action cache keys Pants generates?

Sharing the relevant

pants.ci.toml

pieces as well in case they are helpful

# CI-specific Pants configuration
# Inherits from pants.toml and overrides settings for CI environment
#
# Usage: PANTS_CONFIG_FILES=pants.ci.toml pants test

[GLOBAL]
level = "info"

# Depot Cache: shared remote cache across CI runs (REAPI)
# DEPOT_TOKEN env var is set in pants-ci.yml from GitHub Secrets
remote_cache_read = true
remote_cache_write = true
remote_store_address = "<grpcs://cache.depot.dev>"
remote_cache_warnings = "first_only"
remote_store_headers = { Authorization = "%(env.DEPOT_TOKEN)s" }

# Memory limit for CI environment (96-core, 384GiB larger runner)
process_total_child_memory_usage = "320GiB"

# Parallelism sized for 96 vCPUs
process_execution_local_parallelism = 96

# Enable pantsd daemon for faster subsequent Pants invocations in CI
# Multiple Pants commands (lint, check, test, package) benefit from shared daemon
pantsd = true
pantsd_max_memory_usage = "8GiB"

# Skip tests marked with skip_cicd tag
tag = ["-skip_cicd"]

[stats]
log = true

[test]
use_coverage = false
output = "failed"
report = true

[pytest]
args = [
    "-v",
    "-ra",
    "--tb=short",
    "-p",
    "no:cacheprovider",
    "--asyncio-mode=auto",
    # Deterministic xdist worker count for remote cache key stability.
    # Auto-detection can vary between runners, busting cache keys.
    "-n",
    "8",
]

why does

Run Pytest

always miss remote cache between CI runs when the Process struct appears identical?