curved-television-656809/13/2022, 2:57 PM
from within a ubuntu docker image, it keeps failing for me with:
[...] File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute self._raise_on_error([t for _, t in throws]) File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 507, in _raise_on_error raise ExecutionError( Exception message: 1 Exception encountered: Exception: Error making env for store at "/root/.cache/pants/lmdb_store/files/0": Cannot allocate memory NoneType: None
but even if I select another location for the lmdb store it gives the same kind of error.
./pants --no-pantsd --no-watch-filesystem --no-local-cache version
fast-nail-5540009/13/2022, 2:59 PM
Cannot allocate memory
curved-television-656809/13/2022, 2:59 PM
root@62d0745e8aae:/src# ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31715 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
fast-nail-5540009/13/2022, 3:01 PM
(for "max locked memory") - I think that LMDB uses mmap to map the on-disk files into memory. Unclear if "max locked memory" is related to mmap usage but easy enough to relax that limit.
ulimit -l unlimited
curved-television-656809/13/2022, 3:01 PM
.. likely a technical reason as to why not though.. @witty-crayon-22786?
fast-nail-5540009/13/2022, 3:20 PM
curved-television-656809/13/2022, 3:29 PM
wide-midnight-7859809/13/2022, 3:59 PM
refined-addition-5364409/13/2022, 4:30 PM
hmm… or this is an issue with running an amd64 image on the M1 rather than an arm64 one..This I have seen happening with my colleagues. It gets stuck on “starting pantsd” for long time and then times out Although I didn’t verify if it’s the difference in the architecture which is creating this issue.
curved-television-656809/13/2022, 4:31 PM
Does this problem still occur if you mount the cache on a separate volume?Tried to point the lmdb_store to other paths, including a volume mounted one, no avail 😕
freezing-lamp-1212309/13/2022, 4:34 PM
witty-crayon-2278609/13/2022, 4:35 PM
settings lower (particularly https://www.pantsbuild.org/docs/reference-global#local_store_files_max_size_bytes) will help.
would’ve been nice if it didn’t have to create the lmdb store at all when usingeven when the cache is disable, the store is used to snapshot files so that we can then hold.. likely a technical reason as to why not though.. @witty-crayon-22786?
curved-television-656809/13/2022, 6:05 PM
how much memory is available/allocated to the container?
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS d3bfea806e90 crazy_davinci 0.00% 9.855MiB / 7.752GiB 0.12% 806B / 0B 0B / 0B 2
it’s possible that adjusting theYou’re an absolute ace, @witty-crayon-22786! 🙌 ❤️ For posterity, here’s my settings:settings lower will help
That is I dropped the 16GB ones to 1.6GB and the 256GB down to 2.56GB or there abouts..
export PANTS_LOCAL_STORE_PROCESSES_MAX_SIZE_BYTES=1600000000 PANTS_LOCAL_STORE_FILES_MAX_SIZE_BYTES=2560000000 PANTS_LOCAL_STORE_DIRECTORIES_MAX_SIZE_BYTES=1600000000 export PANTS_WATCH_FILESYSTEM=false PANTS_LOCAL_CACHE=false PANTS_PANTSD=false
witty-crayon-2278609/13/2022, 6:21 PM
curved-television-656809/13/2022, 6:23 PM
witty-crayon-2278609/13/2022, 6:23 PM
hundreds-father-40409/13/2022, 6:23 PM
LMDB’s time is definitely limited.What comes next?
curved-television-656809/13/2022, 6:24 PM
might also want to adjust the shard countthere’s a lot of tweakability (for better and worse) here - come to think of “profiles”, like slim/local/ci/custom/… 😉
witty-crayon-2278609/13/2022, 6:24 PM
What comes next?unclear. sqlite, rocksdb, who knows