I'm a bit lost on how to improve CI pipeline performance. I have a CI pipeline which first runs something like:

pants \
  --changed-since=origin/main \
  --changed-dependents=transitive \
  --filter-target-type=docker_image
  list

Then I dynamically generate one CI job per

docker_image

target to run `pants package`/`pants publish` against, etc. The issue is that the

pants list

command takes about 2 minutes in our monorepo. I was hoping to figure out some trick to reduce the time it takes to generate a list of changed

docker_image

targets. Does anyone have any creative workarounds? I am already using a remote cache (bazel-remote-cache) but it doesn't seem to help. pants version: 2.24.1

pants list

taking such a long time is quite surprising to me, as it does very little work (certainly not much interesting in terms of dependency inference or running external processes) To make the debugging faster, can you reproduce the slowness locally?

How large is your codebase? I do similar checks in multiple locations in our CI, and it's not even noticeable. What languages are involved?

9k python source files and about 100 docker_image targets.

Does this change significantly depending on what flags you provide?

$ hyperfine  \
    --setup 'pkill pantsd' \
    --warmup 1 \
    'pants list --changed-since=origin/main' \
    'pants list --changed-since=origin/main --changed-dependents=transitive' \
    'pants list --changed-since=origin/main --filter-target-type=docker_image' \
    'pants list --changed-since=origin/main --changed-dependents=transitive --filter-target-type=docker_image'

For me, it gives the following results:

Benchmark 1: pants list --changed-since=origin/main
  Time (mean ± σ):     744.8 ms ±  54.2 ms    [User: 1.3 ms, System: 17.8 ms]
  Range (min … max):   691.3 ms … 857.7 ms    10 runs

Benchmark 2: pants list --changed-since=origin/main --changed-dependents=transitive
  Time (mean ± σ):     868.6 ms ±  58.8 ms    [User: 6.0 ms, System: 27.5 ms]
  Range (min … max):   813.0 ms … 1027.1 ms    10 runs

Benchmark 3: pants list --changed-since=origin/main --filter-target-type=docker_image
  Time (mean ± σ):     708.3 ms ±  38.3 ms    [User: 5.5 ms, System: 12.6 ms]
  Range (min … max):   667.1 ms … 787.6 ms    10 runs

Benchmark 4: pants list --changed-since=origin/main --changed-dependents=transitive --filter-target-type=docker_image
  Time (mean ± σ):     863.3 ms ±  59.1 ms    [User: 4.1 ms, System: 12.2 ms]
  Range (min … max):   811.9 ms … 1022.5 ms    10 runs

Summary
  pants list --changed-since=origin/main --filter-target-type=docker_image ran
    1.05 ± 0.10 times faster than pants list --changed-since=origin/main
    1.22 ± 0.11 times faster than pants list --changed-since=origin/main --changed-dependents=transitive --filter-target-type=docker_image
    1.23 ± 0.11 times faster than pants list --changed-since=origin/main --changed-dependents=transitive

So one thing to know... We have 100+ python source roots with about 7 resolves. Many of the python packages/source roots are in 2 or more resolves.

Yeah, but my stats is ~1k sources, a dozen containers, and ~100 reqs.

It went down from 27s to 24s (eye balling aka watching the counter)

Ok, still sizeable. How many direct changes are you running this test on?

I'm doing a --changed-since=HEAD~250 for testing

No, that's plenty clear 😛 I'm reading the Pants source, and one could imagine inverting the queries, so instead of starting from changed files and finding all dependents, we find all targets of a specific type and see if they depend on the changed files. I think for this specific query it would be hugely beneficial. That would avoid the dependents-mapping step.

ahh, yes that does seem like a more efficient approach.

The way it's done for --dependents=transitive is only used by the

changed

backend, as far as I can tell. Dependencies, on the other hand, is queried everywhere. So the question is just how the scaling works out.

So another thought/question: Is there a way to tell pants to ignore an entire tree of the monorepo? In our case we have very distinct parts of our monorepo and some areas do not need to be considered when checking for these changes in docker targets.

https://github.com/pantsbuild/pants/issues/14243,

--filter-address-regex

in theory

@gorgeous-winter-99296 Thanks for the back and forth. Sad there is no solution for repositories with such high target counts.

It's an interesting point. I wonder if this code could be migrated. @happy-kitchen-89482 might've already thought about the specs handling/filtering, he's on a long-running quest to move options handling etc to Rust. The specific rule you mentioned above is here: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/project_info/dependents.py#L42-L65, for example. That is coming from here: https://github.com/pantsbuild/pants/blob/b7f08e9926d84f201f87e56dd643e105de5f1c4b/src/python/pants/vcs/changed.py#L70-L100 That block is the major diff caused by --dependents=transitive.

Does anyone have any creative workarounds?

To clarify since you mentioned you can reproduce locally, does having

pantsd

effect performance? If you do

pants list ::

and then the changed calculation, is there any change in performance? Wild Speculation: Does the new rust Dockerfile parser help at all?

@curved-manchester-66006 The new rust dockerfile-parser did not impact performance.

$ hyperfine  \
    --warmup 1 --runs 2 --setup 'pkill pantsd ||:' \
    --prepare 'pants list //::' 'pants list --changed-since=HEAD~250' \
    --prepare 'pants list //::' 'pants list --changed-since=HEAD~250 --changed-dependents=transitive' \
    --prepare 'pants list //::' 'pants list --changed-since=HEAD~250 --filter-target-type=docker_image' \
    --prepare 'pants list //::' 'pants list --changed-since=HEAD~250 --changed-dependents=transitive --filter-target-type=docker_image'
Benchmark 1: pants list --changed-since=HEAD~250
  Time (mean ± σ):     11.909 s ±  0.229 s    [User: 0.035 s, System: 0.081 s]
  Range (min … max):   11.747 s … 12.071 s    2 runs

Benchmark 2: pants list --changed-since=HEAD~250 --changed-dependents=transitive
  Time (mean ± σ):     51.341 s ±  0.212 s    [User: 0.098 s, System: 0.276 s]
  Range (min … max):   51.191 s … 51.491 s    2 runs

Benchmark 3: pants list --changed-since=HEAD~250 --filter-target-type=docker_image
  Time (mean ± σ):     11.730 s ±  0.003 s    [User: 0.020 s, System: 0.025 s]
  Range (min … max):   11.729 s … 11.732 s    2 runs

Benchmark 4: pants list --changed-since=HEAD~250 --changed-dependents=transitive --filter-target-type=docker_image
  Time (mean ± σ):     50.657 s ±  0.183 s    [User: 0.018 s, System: 0.024 s]
  Range (min … max):   50.527 s … 50.787 s    2 runs

Summary
  pants list --changed-since=HEAD~250 --filter-target-type=docker_image ran
    1.02 ± 0.02 times faster than pants list --changed-since=HEAD~250
    4.32 ± 0.02 times faster than pants list --changed-since=HEAD~250 --changed-dependents=transitive --filter-target-type=docker_image
    4.38 ± 0.02 times faster than pants list --changed-since=HEAD~250 --changed-dependents=transitive