Pants started misbehaving today, any clues? ```12:...
# general
l
Pants started misbehaving today, any clues?
Copy code
12:48:02.56 [INFO] Initializing scheduler...
12:48:02.79 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 123, in main
    PantsLoader.main()
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 110, in main
    cls.run_default_entrypoint()
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 92, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 123, in run
    return self._connect_and_execute(pantsd_handle, executor, start_time)
  File "/Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/venvs/2.16.0rc1/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 165, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). If you keep seeing this error message, try the troubleshooting steps below. If none of th
ose help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
 - Exit other applications, including applications running in the background.
 - Set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`).
 - Disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.
1
w
is there any information in
.pants.d/pants.log
?
l
Copy code
12:48:02.31 [INFO] pantsd 2.16.0rc1 running with PID: 44444
12:48:02.31 [INFO] handling request: `--python-repos-find-links=-['file:///Users/theoribeiro/Library/Caches/nce/142382d6e3ac0960e7d4f81b9df9de9559b7ac698db13d20cbd0e4aa57f68c1d/bindings/find_links/2.16.0rc1/0d86147e/index.html'] export-codegen //node/apps/checkout: --keep-sandboxes=on_failure`
/Users/gha/.cargo/git/checkouts/lmdb-rs-369bfd26153a2575/6ae7a55/lmdb-sys/lmdb/libraries/liblmdb/mdb:3154: Assertion 'pglast <= env->me_pglast' failed in mdb_freelist_save()
w
Copy code
Assertion 'pglast <= env->me_pglast' failed in mdb_freelist_save()
yuck. that would be a bug in LMDB.
can resolve it by clearing
~/.cache/pants/lmdb_store
… sorry for the trouble.
l
Ah, perfect, I’ll try that. Don’t know if it helps to diagnose the issue but right before I got stuck with this error this happened:
Copy code
IntrinsicError: Bytes from stdout: Got hash collision reading from store - digest Digest { hash: Fingerprint<0baca2fec1af96b3b88cae6522fad455ea890e979c1fd0da5e0057bc233760fa>, size_bytes: 10 } was requested, but retrieved bytes with
 that fingerprint had length 8. Congratulations, you may have broken sha256! Underlying bytes: [237, 111, 73, 100, 0, 0, 0, 0]
💥 2
🍾 1
b
That's an incredibly short collision!
w
uh… yea. not sure i believe that, heh. seems more likely to have been an LMDB bug given the ensuing corruption.
👍 1
Copy code
>>> hashlib.sha256(bytes([237, 111, 73, 100, 0, 0, 0, 0])).hexdigest()
'5ba1255b9e651711455b5e21da842c6c2f3a10fdd6d78f96cc67ea26dcc34815'
…?
also, it’s probably not fair to assume “LMDB bug”… at least some of these might actually be hardware/disk issues
g
@brief-ambulance-96324 we also see this issue