Hi I am running `shell command`s that internally execute som Pants #general

Hi! I am running `shell_command`s that internally ...

melodic-agent-3403

10/15/2024, 3:42 PM

Hi! I am running `shell_command`s that internally execute some simple read-only

git

commands. Pants likes to frequently cancel this command (this thread indicates that it's probably due to its speculative execution). However, the way it cancels shell commands seems to sometimes kills

git

in the middle, leaving dangling

.git/index.lock

files that block any subsequent git commands. This leads me to believe that pants cancels commands using

SIGKILL

, which seems unreasonably violent. Is this true and is there a way to stop it, or work around it so git can clean up after itself?

melodic-agent-3403

10/15/2024, 3:44 PM

(FWIW, the entire shell_command takes about 50ms to execute, during which it calls git 6 times, so I am surprised it even manages to be cancelled so often)

fast-nail-55400

10/15/2024, 4:02 PM

Pants has some (unused) infrastructure to gracefully kill processes but which the local execution code does not configure. See https://github.com/pantsbuild/pants/blob/276f3d43f18c6382639e632f6a24b9b59a849d41/src/rust/engine/process_execution/src/local.rs#L313 (that second parameter is the graceful kill timeout which is not configured).

fast-nail-55400

10/15/2024, 4:05 PM

So it would need a source change to fix, but the fix most likely just involves creating configuration for the timeout and flowing it through to the call site where it is passed to the execution code.

melodic-agent-3403

10/15/2024, 4:27 PM

Thanks for digging into it! Do you know of a way to work around that until then? Make sure

git

survives the parent process being killed, while still being able to read its output? I can wrap my script into another script that is spawned as some subprocess. It's ugly, but maybe necessary 😞

melodic-agent-3403

11/12/2024, 3:12 PM

@fast-nail-55400 This is still a large problem for us. I have tried everything i could think of to spawn

git

as a subprocess that is safe from pants' SIGKILL, but nothing seems to work: •

to run it as a background process •

disown

to detach it from the shell's jobs •

nohup

to detach it from the shell's pipes •

bash -c "<subcommand> &"

to make sure its process parent ID gets set to 1 Several of these work fine if i run them manually and

kill -9 <pid>

their calling bash script. But somehow pants still finds my orphans and kills them, leaving behind occasional git lockfiles and corrupting our users' repo. Any ideas to fix this?

4 Views

Open in Slack

Previous Next