```# Performance Investigation: --changed-dependents=transitive on 53K targets

## The Problem

pants --changed-since=<ref> --changed-dependents=transitive --filter-target-type=docker_image list takes ~3.5 minutes in our monorepo (53K targets), regardless of how few files actually changed. This is ~30-40% of our build pipeline time.

## Root Cause

The bottleneck is map_addresses_to_dependents() in src/python/pants/backend/project_info/dependents.py. When --changed-dependents is used, this rule must build the full reverse dependency graph by calling resolve_dependencies() for every

target in the repo:

@rule(desc="Map all targets to their dependents")

async def map_addresses_to_dependents(all_targets: AllUnexpandedTargets) -> AddressToDependents:

dependencies_per_target = await concurrently(

resolve_dependencies(DependenciesRequest(tgt.get(Dependencies), ...))

for tgt in all_targets # ALL 53K targets

)

This is expensive because each resolve_dependencies() call runs dependency inference (Python import parsing, Docker COPY analysis, etc.). With 53K targets, this takes ~150 seconds of the ~210 second total.

## Why Caching Doesn't Help

- pantsd in-memory cache: AllUnexpandedTargets is https://github.com/pantsbuild/pants/blob/main/src/python/pants/engine/target.py#L1023-L1025. Any filesystem change invalidates it, which cascades to invalidate AddressToDependents. We benchmarked identical commands on warm pantsd — still ~3 minutes each time.

- Remote cache: Only caches process execution results (test runs, compilation), not rule engine computations like dependency inference.

- LMDB local store: Same — only process results, not rule outputs.

## What We Tried

┌─────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────┐

│ Approach │ Result │

├─────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────┤

│ Warm pantsd (identical command twice) │ 3m00s → 3m01s (no improvement) │

├─────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────┤

│ Forward BFS from filtered targets only (https://github.com/pantsbuild/pants/pull/23224) │ 3m39s → 2m42s (26% faster, resolves 24K instead of 53K targets) │

├─────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────┤

│ Persistent dep cache on disk (https://github.com/pantsbuild/pants/pull/23228) │ 3m22s → 43s (5x faster) │

└─────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────┘

## Our Solution: Persistent Dependency Inference Cache

https://github.com/pantsbuild/pants/pull/23228 adds an opt-in --incremental-dependents-enabled flag that persists the forward dependency graph to ~/.cache/pants/incremental_dep_graph_v2.json (29MB, 1.3MB compressed). On subsequent runs, only targets with changed source files (by SHA-256 content hash) have their dependencies re-resolved.

Cold (no cache): 3m22s — 52,927 targets resolved, writes cache

Warm (from cache): 43s — 0 targets resolved, 52,927 from cache

The cache is portable across machines (SHA-256 content hashing, not mtime), so it works on ephemeral CI agents via S3.

## The Architectural Gap

The fundamental issue is that Pants' rule engine caches rule results in-memory only and has all-or-nothing invalidation — there's no concept of "re-resolve deps for 50 changed targets and reuse the other 52,877 from last run." map_addresses_to_dependents either returns a fully cached result or recomputes from scratch for all targets.

A native solution would be persistent rule-level caching in the engine (extending LMDB to cache resolve_dependencies results per-target, keyed by content hash). Our PR approximates this at the Python level.```

```Profiling Data: map_addresses_to_dependents with 53K Targets

I instrumented resolve_dependencies and map_addresses_to_dependents on upstream main (2.32.0.dev7) to understand where the time goes. No changes to caching or batching logic — just timing measurements on the stock code.

Setup

- 52,927 targets (20,850 python_source, 15,422 file, 8,620 resource, 2,945 python_test, plus Docker, Shell, etc.)

- Command: pants --no-pantsd --changed-since=HEAD~3 --changed-dependents=transitive --tag="-integration" --filter-target-type="+python_test" filter

- Timing added to each resolve_dependencies call and to map_addresses_to_dependents phases

Results

map_addresses_to_dependents phase breakdown:

- Resolve all dependencies (the concurrently() call): 122.5s

- Build reverse map (pure Python dict construction): 0.1s

So 99.9% of the time is in the concurrently(resolve_dependencies(...) for tgt in all_targets) call.

Per-call resolve_dependencies stats (52,927 calls):

┌───────────────────────┬────────────┐

│ Metric │ Value │

├───────────────────────┼────────────┤

│ p50 (median) │ 13.4s │

├───────────────────────┼────────────┤

│ Average │ 38.4s │

├───────────────────────┼────────────┤

│ p99 │ 112.7s │

├───────────────────────┼────────────┤

│ Max │ 118.2s │

├───────────────────────┼────────────┤

│ Sum of all calls │ 2,030,764s │

├───────────────────────┼────────────┤

│ Wall time │ 122.5s │

├───────────────────────┼────────────┤

│ Effective parallelism │ ~16,500x │

└───────────────────────┴────────────┘

The sum-of-all-durations is ~2 million seconds but wall time is only 122s, which is consistent with ~16K+ calls in flight concurrently (as expected from try_join_all on 53K futures).

The per-call latency grows over the course of the run:

┌─────────────────┬──────────────┬──────────────┐

│ Progress │ Avg per call │ p50 per call │

├─────────────────┼──────────────┼──────────────┤

│ First 10K calls │ 309ms │ 119ms │

├─────────────────┼──────────────┼──────────────┤

│ At 20K │ 219ms │ 120ms │

├─────────────────┼──────────────┼──────────────┤

│ At 30K │ 3,564ms │ 139ms │

├─────────────────┼──────────────┼──────────────┤

│ At 40K │ 21,571ms │ 178ms │

├─────────────────┼──────────────┼──────────────┤

│ At 50K │ 34,824ms │ 382ms │

└─────────────────┴──────────────┴──────────────┘

The median stays relatively low (119ms → 382ms), but the average and tail latency explode as more calls are in flight — calls submitted later wait for the full duration of all earlier calls.

Batching experiment

I also tested whether batching the concurrently() call helps reduce contention:

┌────────────────────────┬───────────┐

│ Batch Size │ Wall Time │

├────────────────────────┼───────────┤

│ 53K (unbatched, stock) │ 2m40s │

├────────────────────────┼───────────┤

│ 5,000 │ 2m44s │

├────────────────────────┼───────────┤

│ 500 │ 2m38s │

├────────────────────────┼───────────┤

│ 50 │ 2m31s │

├────────────────────────┼───────────┤

│ 1 (fully sequential) │ 2m27s │

└────────────────────────┴───────────┘

Batching at the Python level doesn't meaningfully help. The overhead is below the concurrently() call — inside the engine's task scheduling, graph node management, and Python↔Rust generator protocol for each of the 53K calls.

Interpretation

The per-call work is fast individually (p50 ~120ms), but 53K calls through the engine's concurrent task infrastructure accumulates significant overhead. The reverse map construction itself is trivial (0.1s). The entire cost is the engine processing 53K resolve_dependencies calls, even when the underlying dep inference results are cached.

I don't have a theory for what specifically in the engine is slow at this scale — it could be task scheduling overhead, the generator send/receive protocol, workunit management, or something else in the Rust engine. But the data clearly shows the cost is in the aggregate overhead of 53K concurrent rule calls, not in the dep inference computation itself.```

``` I instrumented the Rust engine (src/rust/engine/src/nodes/task.rs generate() function) on upstream main (2.32.0.dev7) to trace what actually happens during the map_addresses_to_dependents call on our 53K-target repo. Here's what I found.

It's not 53K operations — it's 1.86 million.

Each of the 52,927 resolve_dependencies calls is a Python async generator that spawns nested generators for resolve_target, determine_explicitly_provided_dependencies, infer_dependencies (per inference type), resolve_target_parametrizations,

validate_dependencies, etc. The Rust engine executes each as a separate generator via gen_generator().

Top-level resolve_dependencies calls: 52,927

Total generators executed: 1,860,000 (~35 per resolve_dependencies)

Avg loop iterations per generator: 2.2

Total generator_send() calls: ~4,100,000 (each acquires GIL via Python::attach)

Wall time: 111.1s

Cost per GIL round-trip: ~27µs

27µs per round-trip is reasonable for Python↔Rust FFI. The problem is pure volume: 4.1 million Python::attach() → generator.send() → parse response → dispatch cycles.

Batching the top-level concurrently() at the Python level doesn't help — I tested batch sizes of 53K, 5K, 500, 50, and 1 (fully sequential). All within noise (~2m30-2m44s). This makes sense: the bottleneck isn't the 53K concurrent futures

contending on something. It's the 1.86M nested generators those futures spawn, regardless of how you batch the outer call.

Batch Size Wall Time

53K (stock) 2m40s

5,000 2m44s

500 2m38s

50 2m31s

1 (seq) 2m27s

The per-call latency distribution confirms this:

Progress Avg/call p50/call

10K calls 309ms 119ms

20K 219ms 120ms

30K 3,564ms 139ms

40K 21,571ms 178ms

50K 34,824ms 382ms

The p50 stays low (individual calls are fast), but the average explodes as later calls include wait time for all concurrent work. The sum of all per-call wall times is ~2M seconds across 111s of wall time — consistent with ~16,500 calls

in-flight simultaneously.

Where the time goes:

- resolve_all (the concurrently() call): 111.1s

- build_reverse (pure Python dict construction): 0.1s

The graph cache IS working — entry.get_node_result() returns Completed results via future::ready() for cached nodes. But even for a cached result, the engine still: creates a Python coroutine → sends to it → gets back a rule call → looks up

the graph node → finds it cached → returns the value → sends back to the coroutine → which yields the final result. That's the 27µs overhead, and it happens 4.1M times.

The fix would need to reduce the number of generator round-trips for cached results — either by short-circuiting in the Rust generate() function when a node is already cached, or by porting the hot-path rules to native Rust to eliminate the

Python generator protocol entirely.```

test