Issue with running pants in a ubuntu container image thread

Question

curved-television-6568 · Answer

Hi Trying to run ` pants version` from within a ubuntu docker image it keeps failing for me with ``` File root cache pants setup bootstrap Linux x86 64 2 13 0 py38 lib python3 8 site packages pants engine internals scheduler py line 521 in execute self raise on error t for t in throws File root cache pants setup bootstrap Linux x86 64 2 13 0 py38 lib python3 8 site packages pants engine internals scheduler py line 507 in raise on error raise ExecutionError Exception message 1 Exception encountered Exception Error making env for store at root cache pants lmdb store files 0 Cannot allocate memory NoneType None```

curved-television-6568 · Answer

I m invoking it thus ``` pants no pantsd no watch filesystem no local cache version``` but even if I select another location for the lmdb store it gives the same kind of error

fast-nail-55400 · Answer

What are the ulimit s

fast-nail-55400 · Answer

since `Cannot allocate memory`

curved-television-6568 · Answer

```root 62d0745e8aae src ulimit a core file size blocks c unlimited data seg size kbytes d unlimited scheduling priority e 0 file size blocks f unlimited pending signals i 31715 max locked memory kbytes l 64 max memory size kbytes m unlimited open files n 1048576 pipe size 512 bytes p 8 POSIX message queues bytes q 819200 real time priority r 0 stack size kbytes s 8192 cpu time seconds t unlimited max user processes u unlimited virtual memory kbytes v unlimited file locks x unlimited```

curved-television-6568 · Answer

not sure if this affects things but running from a Mac M1 using rancher desktop for the docker stuff

fast-nail-55400 · Answer

maybe set `ulimit l unlimited` for max locked memory I think that LMDB uses mmap to map the on disk files into memory Unclear if max locked memory is related to mmap usage but easy enough to relax that limit

curved-television-6568 · Answer

I ll try it slightly smiling face

curved-television-6568 · Answer

ahem operation not permitted so I suspect this could be a perm issue where I need to provide more privs to running the container perhaps

curved-television-6568 · Answer

OK nope adding all capabilities and changing the l ulimit to unlimited had unfortunately no effect

curved-television-6568 · Answer

would ve been nice if it didn t have to create the lmdb store at all when using ` no local cache` likely a technical reason as to why not though < witty crayon 22786>

fast-nail-55400 · Answer

Where is the container running rancher sounds like a locally installed Kubernetes

curved-television-6568 · Answer

no rancher desktop is just like docker desktop but from rancher with some builtin features for setting up a k8s cluster not using that so it s a regular docker client server setup on a my machine

curved-television-6568 · Answer

BUT I m not sure how the docker setup is configured compared to that you get with docker desktop

curved-television-6568 · Answer

hmm or this is an issue with running an amd64 image on the M1 rather than an arm64 one

curved-television-6568 · Answer

but I don t get it really as there s qemu virtualization going on oh well

wide-midnight-78598 · Answer

Does this problem still occur if you mount the cache on a separate volume I vaguely recall playing around with that when I ran into this type of problem way back

refined-addition-53644 · Answer

gt hmm or this is an issue with running an amd64 image on the M1 rather than an arm64 one This I have seen happening with my colleagues It gets stuck on starting pantsd for long time and then times out Although I didn t verify if it s the difference in the architecture which is creating this issue

curved-television-6568 · Answer

gt Does this problem still occur if you mount the cache on a separate volume Tried to point the lmdb store to other paths including a volume mounted one no avail confused

freezing-lamp-12123 · Answer

how much memory is available allocated to the container

witty-crayon-22786 · Answer

it s possible that adjusting the `local store max size bytes` settings lower particularly <https www pantsbuild org docs reference global local store files max size bytes> will help

witty-crayon-22786 · Answer

LMDB will mmap the entire store and that can hit different virtual memory limits in different environments

witty-crayon-22786 · Answer

gt would ve been nice if it didn t have to create the lmdb store at all when using ` no local cache` likely a technical reason as to why not though < witty crayon 22786> even when the cache is disable the store is used to snapshot files so that we can then hold `Digests` for them

curved-television-6568 · Answer

gt how much memory is available allocated to the container ```CONTAINER ID NAME CPU MEM USAGE LIMIT MEM NET I O BLOCK I O PIDS d3bfea806e90 crazy davinci 0 00 9 855MiB 7 752GiB 0 12 806B 0B 0B 0B 2```

curved-television-6568 · Answer

gt it s possible that adjusting the `local store max size bytes` settings lower will help You re an absolute ace < witty crayon 22786> raised hands heart For posterity here s my settings ```export PANTS LOCAL STORE PROCESSES MAX SIZE BYTES=1600000000 PANTS LOCAL STORE FILES MAX SIZE BYTES=2560000000 PANTS LOCAL STORE DIRECTORIES MAX SIZE BYTES=1600000000 export PANTS WATCH FILESYSTEM=false PANTS LOCAL CACHE=false PANTS PANTSD=false``` That is I dropped the 16GB ones to 1 6GB and the 256GB down to 2 56GB or there abouts

witty-crayon-22786 · Answer

LMDB s time is definitely limited it has served us well but has three of four different annoying issues like this

curved-television-6568 · Answer

Thanks also to Tom Shantanu SJ and David for suggestions and questions thumbsup

witty-crayon-22786 · Answer

glad that worked note that you might also want to adjust the shard count to prevent limiting the max size of files

hundreds-father-404 · Answer

gt LMDB s time is definitely limited What comes next

curved-television-6568 · Answer

gt might also want to adjust the shard count there s a lot of tweakability for better and worse here come to think of profiles like slim local ci custom wink

witty-crayon-22786 · Answer

gt What comes next unclear sqlite rocksdb who knows