Issue with running pants in a ubuntu container ima...
# general
c
Issue with running pants in a ubuntu container image. 🧵
1
Hi, Trying to run
./pants version
from within a ubuntu docker image, it keeps failing for me with:
Copy code
[...]
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 507, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  Exception: Error making env for store at "/root/.cache/pants/lmdb_store/files/0": Cannot allocate memory

NoneType: None
I’m invoking it thus:
Copy code
./pants --no-pantsd --no-watch-filesystem --no-local-cache version
but even if I select another location for the lmdb store it gives the same kind of error.
f
What are the ulimit's?
(since
Cannot allocate memory
)
c
Copy code
root@62d0745e8aae:/src# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31715
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
not sure if this affects things, but running from a Mac M1 using rancher desktop for the docker stuff.
f
maybe set
ulimit -l unlimited
(for "max locked memory") - I think that LMDB uses mmap to map the on-disk files into memory. Unclear if "max locked memory" is related to mmap usage but easy enough to relax that limit.
c
I’ll try it 🙂
ahem… operation not permitted… so I suspect this could be a perm issue where I need to provide more privs to running the container perhaps.
OK, nope, adding all capabilities and changing the -l ulimit to unlimited had unfortunately no effect..
would’ve been nice if it didn’t have to create the lmdb store at all when using
--no-local-cache
.. likely a technical reason as to why not though.. @witty-crayon-22786?
f
Where is the container running? "rancher" sounds like a locally-installed Kubernetes?
c
no, “rancher desktop” is just like “docker desktop” but from rancher with some builtin features for setting up a k8s cluster (not using that).. so it’s a regular docker client/server setup on a my machine.
BUT, I’m not sure how the docker setup is configured, compared to that you get with docker desktop…
hmm… or this is an issue with running an amd64 image on the M1 rather than an arm64 one..
but I don’t get it, really, as there’s qemu virtualization going on… oh well
w
Does this problem still occur if you mount the cache on a separate volume? I vaguely recall playing around with that when I ran into this type of problem way back
r
hmm… or this is an issue with running an amd64 image on the M1 rather than an arm64 one..
This I have seen happening with my colleagues. It gets stuck on “starting pantsd” for long time and then times out Although I didn’t verify if it’s the difference in the architecture which is creating this issue.
c
Does this problem still occur if you mount the cache on a separate volume?
Tried to point the lmdb_store to other paths, including a volume mounted one, no avail 😕
😕 1
f
how much memory is available/allocated to the container?
w
it’s possible that adjusting the
local_store_*_max_size_bytes
settings lower (particularly https://www.pantsbuild.org/docs/reference-global#local_store_files_max_size_bytes) will help.
💯 1
☝️ 1
LMDB will mmap the entire store, and that can hit different virtual memory limits in different environments.
would’ve been nice if it didn’t have to create the lmdb store at all when using
--no-local-cache
.. likely a technical reason as to why not though.. @witty-crayon-22786?
even when the cache is disable, the store is used to snapshot files so that we can then hold
Digests
for them
1
c
how much memory is available/allocated to the container?
Copy code
CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O     BLOCK I/O   PIDS
d3bfea806e90   crazy_davinci   0.00%     9.855MiB / 7.752GiB   0.12%     806B / 0B   0B / 0B     2
it’s possible that adjusting the
local_store_*_max_size_bytes
settings lower will help
You’re an absolute ace, @witty-crayon-22786! 🙌 ❤️ For posterity, here’s my settings:
Copy code
export PANTS_LOCAL_STORE_PROCESSES_MAX_SIZE_BYTES=1600000000 PANTS_LOCAL_STORE_FILES_MAX_SIZE_BYTES=2560000000 PANTS_LOCAL_STORE_DIRECTORIES_MAX_SIZE_BYTES=1600000000
export PANTS_WATCH_FILESYSTEM=false PANTS_LOCAL_CACHE=false PANTS_PANTSD=false
That is I dropped the 16GB ones to 1.6GB and the 256GB down to 2.56GB or there abouts..
❤️ 1
w
LMDB’s time is definitely limited. it has served us … “well”, but has three of four different annoying issues like this.
👍 1
c
Thanks also to Tom, Shantanu, SJ and David for suggestions and questions 👍
w
glad that worked. note that you might also want to adjust the shard count to prevent limiting the max size of files
🙏 1
h
LMDB’s time is definitely limited.
What comes next?
c
might also want to adjust the shard count
there’s a lot of tweakability (for better and worse) here - come to think of “profiles”, like slim/local/ci/custom/… 😉
w
What comes next?
unclear. sqlite, rocksdb, who knows