Hi all, I'm encountering an issue whilst doing

./pants package ...

which either ends in one of two scenarios: 1. Running with

./pants package

results in a nasty traceback stating that the pantsd process was killed during the run

Traceback (most recent call last):
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 90, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 117, in run
    return self._connect_and_execute(pantsd_handle, start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 151, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). You can set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`). You can also disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

If neither of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).

Re-running with

./pants --pantsd-max-memory-usage=8GiB package ...

had no affect on the outcome however and running with

/usr/bin/time -v

prefixing the aforementioned command gives an output which suggests a maximum of ~48MB was used (

free -h

suggests there is

10Gi

memory available on my WSL Ubuntu system). 2. Running with

./pants --no-pantsd package ...

(as suggested by the error message from point 1) results in different message:

/github/home/.cargo/git/checkouts/lmdb-rs-369bfd26153a2575/6ae7a55/lmdb-sys/lmdb/libraries/liblmdb/m:2432: Assertion 'mp->mp_pgno != pgno' failed in mdb_page_touch()
Aborted

Does anyone have experience with the above type of problem? How would I go about debugging the root cause behind the issue? Any help would be most appreciated 🙏 😃

How are you installing Pants? The 1st backtrace looks like a normal Pants install (

/home/liam/...

) but the second (

/github/home/.cargo/...

) makes it look like you've cloned the Pants repo - or something. Very weird path for that error message.

Thanks @enough-analyst-54434 it looks like clearing the cache (by moving it elsewhere) solved my issue!

Hrm. Never satisfying. I'm glad you're problem is solved though. How big is the lmdb_store subdirectory of the saved cache?

Thanks for the suggestion though - it got me out of a loop of frustration and pain 🙌 Turns out I have about 15GB of

.cache/pants

which I guess explains the OOM error?

They are not 1-1. The lmdb is memory-mapped; so even if your complete build history is 15GB, it seems highly unlikely the current build requires all of that 15GB and thus must page it all in.

@gray-shoe-19951 this seems similar to what we are experiencing