Hi all! I've been using pants for months without issue, but suddenly today I've started getting errors on every pants command due to the process being killed during its run. I'm running pants on WSL.

$ pants check ::
12:03:34.86 [INFO] Initializing scheduler...
12:03:38.80 [INFO] Scheduler initialized.
⠲ 0.84s Resolve coarsened targets
⠒
⠒
⠒
⠒
⠒
⠒
⠒
Traceback (most recent call last):
  File "/home/mcandela/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.19.0/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/mcandela/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.19.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/home/mcandela/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.19.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/home/mcandela/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.19.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 91, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/mcandela/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.19.0/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 132, in run
    return remote_runner.run(start_time)
  File "/home/mcandela/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.19.0/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 123, in run
    return self._connect_and_execute(pantsd_handle, executor, start_time)
  File "/home/mcandela/.cache/nce/60b513559c7b53eb2acecbd7b8aaaeb942686f3997d07fa77377b51324f58fda/bindings/venvs/2.19.0/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 186, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). If you keep seeing this error message, try the troubleshooting steps below. If none of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
 - Exit other applications, including applications running in the background.
 - Set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`).
 - Disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

The error message points to it being a memory issue, though the task manager / htop shows free memory not being used. I've tried running after a fresh reboot, setting a lower

--pantsd-max-memory-usage

and got the same error. Running with

--no-pantsd

results in a different error

$ pants check :: --no-pantsd
⠴ 0.96s Resolve coarsened targets
⠲
⠲
⠲
⠲
⠲
⠲
⠲
/github/home/.cargo/git/checkouts/lmdb-rs-369bfd26153a2575/6ae7a55/lmdb-sys/lmdb/libraries/liblmdb/m:2126: Assertion 'rc == 0' failed in mdb_page_dirty()
Aborted

Everything was working smoothly a couple days ago, so I'm not sure what I could've changed since then that would cause this issue. I've tried upgrading to the latest pants version to see if there were any stuck config from a failed run, but same error. What would be the best way of doing a fresh install to see if the issue is there? Thanks!

If the size of the various caches could be an issue.. clearing those might help. See https://www.pantsbuild.org/2.20/docs/using-pants/using-pants-in-ci#directories-to-cache for some pointers about cache directories.

I'll give that a go, thanks!

Any other suggestions? I’ve run into the same issue and

rm -rf ~/.cache/pants/

does not help 😞

another cache dir that holds state that could be worth trying to reset is the

nce

cache (

/home/mcandela/.cache/nce/

from the logs in this thread)

Tried that too but no luck. In my case the directory in the logs was

/Users/narayan/Library/Caches/nce/