curved-television-6568
09/13/2022, 2:57 PM./pants version
from within a ubuntu docker image, it keeps failing for me with:
[...]
File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
self._raise_on_error([t for _, t in throws])
File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 507, in _raise_on_error
raise ExecutionError(
Exception message: 1 Exception encountered:
Exception: Error making env for store at "/root/.cache/pants/lmdb_store/files/0": Cannot allocate memory
NoneType: None
./pants --no-pantsd --no-watch-filesystem --no-local-cache version
but even if I select another location for the lmdb store it gives the same kind of error.fast-nail-55400
09/13/2022, 2:59 PMCannot allocate memory
)curved-television-6568
09/13/2022, 2:59 PMroot@62d0745e8aae:/src# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31715
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
fast-nail-55400
09/13/2022, 3:01 PMulimit -l unlimited
(for "max locked memory") - I think that LMDB uses mmap to map the on-disk files into memory. Unclear if "max locked memory" is related to mmap usage but easy enough to relax that limit.curved-television-6568
09/13/2022, 3:01 PM--no-local-cache
.. likely a technical reason as to why not though.. @witty-crayon-22786?fast-nail-55400
09/13/2022, 3:20 PMcurved-television-6568
09/13/2022, 3:29 PMwide-midnight-78598
09/13/2022, 3:59 PMrefined-addition-53644
09/13/2022, 4:30 PMhmm⦠or this is an issue with running an amd64 image on the M1 rather than an arm64 one..This I have seen happening with my colleagues. It gets stuck on āstarting pantsdā for long time and then times out Although I didnāt verify if itās the difference in the architecture which is creating this issue.
curved-television-6568
09/13/2022, 4:31 PMDoes this problem still occur if you mount the cache on a separate volume?Tried to point the lmdb_store to other paths, including a volume mounted one, no avail š
freezing-lamp-12123
09/13/2022, 4:34 PMwitty-crayon-22786
09/13/2022, 4:35 PMlocal_store_*_max_size_bytes
settings lower (particularly https://www.pantsbuild.org/docs/reference-global#local_store_files_max_size_bytes) will help.wouldāve been nice if it didnāt have to create the lmdb store at all when usingeven when the cache is disable, the store is used to snapshot files so that we can then hold.. likely a technical reason as to why not though.. @witty-crayon-22786?--no-local-cache
Digests
for themcurved-television-6568
09/13/2022, 6:05 PMhow much memory is available/allocated to the container?
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
d3bfea806e90 crazy_davinci 0.00% 9.855MiB / 7.752GiB 0.12% 806B / 0B 0B / 0B 2
itās possible that adjusting theYouāre an absolute ace, @witty-crayon-22786! š ā¤ļø For posterity, hereās my settings:settings lower will helplocal_store_*_max_size_bytes
export PANTS_LOCAL_STORE_PROCESSES_MAX_SIZE_BYTES=1600000000 PANTS_LOCAL_STORE_FILES_MAX_SIZE_BYTES=2560000000 PANTS_LOCAL_STORE_DIRECTORIES_MAX_SIZE_BYTES=1600000000
export PANTS_WATCH_FILESYSTEM=false PANTS_LOCAL_CACHE=false PANTS_PANTSD=false
That is I dropped the 16GB ones to 1.6GB and the 256GB down to 2.56GB or there abouts..witty-crayon-22786
09/13/2022, 6:21 PMcurved-television-6568
09/13/2022, 6:23 PMwitty-crayon-22786
09/13/2022, 6:23 PMhundreds-father-404
09/13/2022, 6:23 PMLMDBās time is definitely limited.What comes next?
curved-television-6568
09/13/2022, 6:24 PMmight also want to adjust the shard countthereās a lot of tweakability (for better and worse) here - come to think of āprofilesā, like slim/local/ci/custom/⦠š
witty-crayon-22786
09/13/2022, 6:24 PMWhat comes next?unclear. sqlite, rocksdb, who knows