https://pantsbuild.org/ logo
#general
Title
# general
m

melodic-carpenter-39613

01/16/2023, 2:04 PM
Hi all, I'm encountering an issue whilst doing
./pants package ...
which either ends in one of two scenarios: 1. Running with
./pants package
results in a nasty traceback stating that the pantsd process was killed during the run
Copy code
Traceback (most recent call last):
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 90, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 117, in run
    return self._connect_and_execute(pantsd_handle, start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 151, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). You can set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`). You can also disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

If neither of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
Re-running with
./pants --pantsd-max-memory-usage=8GiB package ...
had no affect on the outcome however and running with
/usr/bin/time -v
prefixing the aforementioned command gives an output which suggests a maximum of ~48MB was used (
free -h
suggests there is
10Gi
memory available on my WSL Ubuntu system). 2. Running with
./pants --no-pantsd package ...
(as suggested by the error message from point 1) results in different message:
Copy code
/github/home/.cargo/git/checkouts/lmdb-rs-369bfd26153a2575/6ae7a55/lmdb-sys/lmdb/libraries/liblmdb/m:2432: Assertion 'mp->mp_pgno != pgno' failed in mdb_page_touch()
Aborted
Does anyone have experience with the above type of problem? How would I go about debugging the root cause behind the issue? Any help would be most appreciated 🙏 😃
e

enough-analyst-54434

01/16/2023, 3:21 PM
How are you installing Pants? The 1st backtrace looks like a normal Pants install (
/home/liam/...
) but the second (
/github/home/.cargo/...
) makes it look like you've cloned the Pants repo - or something. Very weird path for that error message.
Perhaps that's a GH action path baked into the debug symbols.
@melodic-carpenter-39613 that's a very weird error. To clear one thing up, despite what the 1st error message recommended, which is useful for debugging, we need to get you to a state where you can use pantsd or else admit defeat. Pants is ~unuseable from a performance perspective, basically by design, without pantsd.
Now I use Pants on WSL as well. This definitely can work. So I'd like you to try as a 1st step
mv ~/.cache/pants ~/.cache/pants.sav
Then try running with
--no-pantsd
again.
m

melodic-carpenter-39613

01/16/2023, 8:01 PM
Thanks @enough-analyst-54434 it looks like clearing the cache (by moving it elsewhere) solved my issue!
e

enough-analyst-54434

01/16/2023, 8:04 PM
Hrm. Never satisfying. I'm glad you're problem is solved though. How big is the lmdb_store subdirectory of the saved cache?
m

melodic-carpenter-39613

01/16/2023, 8:31 PM
Thanks for the suggestion though - it got me out of a loop of frustration and pain 🙌 Turns out I have about 15GB of
.cache/pants
which I guess explains the OOM error?
e

enough-analyst-54434

01/16/2023, 9:06 PM
They are not 1-1. The lmdb is memory-mapped; so even if your complete build history is 15GB, it seems highly unlikely the current build requires all of that 15GB and thus must page it all in.
b

brief-ambulance-96324

05/05/2023, 1:08 AM
@gray-shoe-19951 this seems similar to what we are experiencing
9 Views