cool-easter-32542
06/09/2025, 8:47 PMpants paths on our large monorepo consumes an excessive amount of memory and runs for 10+ minutes without completing. This happens even after filtering to python_source targets. At its peak, the process consumed ~650 GB of RAM on an r7i.metal-48xl instance (1536 GiB total).
Repo stats:
• ~3,400 source targets
• ~16,000 destination targets
• After adding `--filter-target-type=python_source`:
• ~2,000 source targets
• ~6,600 destination targets
Despite that, performance remains extremely slow. There appears to be either a performance bottleneck in the graphing logic or a memory inefficiency.
According to @benjy, the current Python-based graph algorithm is likely the culprit, and could benefit from a rewrite in Rust with more efficient data structures.
Pants version
2.26.0
OS
Amazon Linux 2023
Additional info
Happy to provide more logs or run experiments. This was blocking me from visualizing connections between resolves, which would've helped validate our target structure. Any flags or workarounds to scope this down further would also be appreciated.
pantsbuild/pants