#23463 Local process execution slowdown proportionate to pants process resident set size Issue created by creeser-nvidia Glossary

vfork

refers to some permutation of a

clone

syscall specifying the

CLONE_VFORK|CLONE_VM

flags.

fork

refers to some permutation of a

clone

syscall, without those flags. Summary Pants' local

execute_process()

as-written forces a process startup latency proportional to parent resident set size, due to forcing the "slow" `fork`+`exec` pattern involving duplication of parent page tables for the new child, instead of permitting the "fast" `vfork`+`exec` pattern which elides that duplication.

execute_process

is likely written this way due to, at the time it was written, Tokio not yet exposing a capability to permit

vfork

while meeting the other requirements

execute_process

has (namely ensuring the child is in its own process group). Using a newer Tokio capability fixes this perf issue. Impact

vfork

on my machine takes ~2ms, with similar timing for

fork

of a parent with a small resident set size. At larger parent resident set sizes,

vfork

perf is ~constant, but

fork

perf degrades. E.g., with a ~30G+ Pants process,

fork

takes 100ms+. For large numbers of processes, this translates to significant and ever-worsening slowdown as the build progresses and more rule results are cached. Fixing this issue permits scale-testing builds that were previously taking hours and gradually running ever more slowly, to instead take tens of minutes with a perf behavior irrespective of memory use. More generally, fixing this should have at-worst neutral impact, and otherwise non-trivial local builds are likely to benefit in at least some minor way. Fixing this issue as recommended will also: • eliminate an

unsafe

block, and • address/obsolete open PR #22894. Mechanism

ManagedChild::spawn()

includes logic to ensure the spawned child is in its own process group: pants/src/rust/process_execution/children/src/lib.rs Lines 40 to 46 in</pantsbuild/pants/commit/375c297b65a27d36b307e29154f2417be047857f|375c297> | unsafe { | | ------------------------------------------------------------------------------- | | command.pre_exec(\|| { | | nix:unistd:setsid() | | .map(\|_pgid| ()) | | .map_err(\|e| std:ioError:other(format!("Could not create new pgid: {e}"))) | | }); | | }; |

pre_exec()

closures are run in the child, after

fork

and before

exec

, and preclude use of the

vfork

fast-path, as alluded to per https://docs.rs/tokio/1.52.3/tokio/process/struct.Command.html#method.pre_exec:

"This also means that all resources such as file descriptors and memory-mapped regions got duplicated."

This Pants code was introduced in 8826bf0 (2021-11-19). Since then, Tokio introduced

Command::process_group()

, which permits arranging for the child to be in its own process group, without precluding the

vfork

fast-path. The Tokio documentation alludes to this (https://docs.rs/tokio/1.52.3/tokio/process/struct.Command.html#method.process_group):

"Sets the process group ID (PGID) of the child process. Equivalent to a setpgid call in the child process, but may be more efficient."

This was introduced in Tokio v1.22.0 (2022-11-17) and stabilized in v1.40.0 (2024-08-30), per https://github.com/tokio-rs/tokio/blob/master/tokio/CHANGELOG.md. Pants today uses Tokio v1.51,1, per pants/src/rust/Cargo.toml Line 223 in</pantsbuild/pants/commit/9dbe5253fe307f7bd3b590a6df4480ff2b034a27|9dbe525> | tokio = "1.51.1" | | ---------------- | Note that this Tokio infrastructure is effectively trivial wrappers over the underlying Rust stdlib APIs of the same names. Notes (1)

execute_process

children are configured to run using the sandbox as CWD. Due to this, a sufficiently-recent glibc (>=2.29, 2019-01) is also necessary to ensure the

vfork

fast-path is taken. glibc v2.29 introduces

posix_spawn_file_actions_addchdir_np()

, an efficient mechanism for ensuring the CWD of the child process is in some target directory. If this symbol is not available at runtime, the Rust stdlib implementation will still fall back to the "slow"

fork

path, performing a naive

fork

->

chdir

->

exec

. See also: glibc documentation for `posix_spawn_file_actions_addchdir_np`: https://pubs.opengroup.org/onlinepubs/9799919799/functions/posix_spawn_file_actions_addchdir.html glibc 2.29 announcement, including for `posix_spawn_file_actions_addchdir_np`: https://lists.gnu.org/archive/html/info-gnu/2019-01/msg00018.html (2) The choice for "fast"/"slow" path is ultimately decided in the Rust stdlib implementation for

spawn()

->

posix_spawn()

(Rust) ->

posix_spawn()

(glibc), where glibc

posix_spawn()

is what is ultimately taking the

vfork

fast-path. See: https://man7.org/linux/man-pages/man3/posix_spawn.3.html See: https://github.com/rust-lang/rust/blob/1.96.0/library/std/src/sys/process/unix/unix.rs The Rust stdlib will perform a naive

fork

-> ... ->

exec

instead of using

posix_spawn()

, if various constraints are not met, where: https://github.com/rust-lang/rust/blob/ac68faa20c58cbccd01ee7208bf3b6e93a7d7f96/library/std/src/sys/process/unix/unix.rs#L463 establishes the requirement that

vfork

is only possible when there are no queued

pre_exec()

closures for the spawn (

fork

-> closures* ->

exec

), and where https://github.com/rust-lang/rust/blob/ac68faa20c58cbccd01ee7208bf3b6e93a7d7f96/library/std/src/sys/process/unix/unix.rs#L641 establishes the requirement for a sufficiently-recent glibc with

posix_spawn_file_actions_addchdir_np()

in order to permit

vfork

when the spawned child must be in a target CWD. Concerns The Pants code documents its purpose as ensuring the child is in its own process group, and accomplishes this, but does so by instead ensuring the child is in its own session which in turn ensures its own process group. The proposed fix merely ensures the child is in its own process group, which satisfies both the extant documentation and the intended purpose RE behavior on e.g. ctrl+c, but is technically different behavior than is implemented today. pantsbuild/pants