#22111 Concurrent ruff invocations take far longer to under pants than outside, on macOS Issue created by huonw Describe the bug There appears to be significant overhead to running some processes concurrently under Pants, on some systems. This is visible for something like ruff, where the individual invocations are very short and thus overhead is noticeable. (I think (but am not 100% sure) that this applies to all processes, and ruff just makes the overhead particularly visible by being so fast.) Reproducer: this sets up 16 python files, and runs ruff on them individually

batch_size = 1

(behaviour is similar to our real repo, where ruff ends up split across many batches of ~100 files): #!/bin/bash cd $(mktemp -d) cat > pants.toml <<EOF [GLOBAL] pants_version = "2.24.0" backend_packages = [ "pants.backend.python", "pants.backend.experimental.python.lint.ruff.check", ] [python] interpreter_constraints = ["CPython==3.10.*"] [lint] batch_size = 1 EOF echo "python_sources(name='src')" > BUILD for i in $(seq 1 16); do touch src$i.py done # start scheduler export PANTS_LOCAL_CACHE=false pants version echo "Pants" time pants lint :: echo "Manual: concurrent" pants export --bin=ruff time ( for i in $(seq 1 16); do dist/export/bins/Ruff/ruff-*/ruff check --force-exclude src$i.py & done wait ) echo "Manual: non-concurrent" time dist/export/bins/Ruff/ruff-/ruff check --force-exclude src.py The

real

times for each: • Pants: 7.043s • Manual: concurrent: 0.467s • Manual: non-concurrent: 0.011s That is, a lot of overhead! The pants

pants lint ::

invocation has output like (I've trimmed out the

All checks passed!

output from Ruff, except for the first two, just as an example):

17:20:27.80 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
All checks passed!


17:20:28.32 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
All checks passed!


17:20:28.72 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:29.13 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:29.53 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:29.92 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:30.33 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:30.72 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:31.13 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:31.52 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:31.92 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:32.33 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:32.72 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:33.12 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:33.51 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
17:20:33.90 [INFO] Completed: Lint with `ruff check` - ruff check succeeded.
✓ ruff check succeeded.
real    0m7.043s

Note how each

Completed

statement is spaced by about 0.4s (and, suspiciously, this is consistent). That is, running 16 ruff invocations takes many seconds, because only 2.5 processes finish per second. I've reproduced this on two different ARM64 macOS machines. I've not reproduced this (i.e. observed normal/fast behaviour) on GHA

macos-14

or

macos-15

(#22110), and within a docker container on macOS. Pants version 2.24.0, 2.25.0 OS macOS Additional info N/A pantsbuild/pants