https://pantsbuild.org/ logo
Join Slack
Powered by
Issue with running pants in a ubuntu container ima...
# general
c
Issue with running pants in a ubuntu container image. šŸ§µ
āœ… 1
Hi, Trying to run 
./pants version
from within a ubuntu docker image, it keeps failing for me with:
Copy code
[...]
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 507, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  Exception: Error making env for store at "/root/.cache/pants/lmdb_store/files/0": Cannot allocate memory

NoneType: None
Iā€™m invoking it thus:
Copy code
./pants --no-pantsd --no-watch-filesystem --no-local-cache version
but even if I select another location for the lmdb store it gives the same kind of error.
f
What are the ulimit's?
(since 
Cannot allocate memory
)
c
Copy code
root@62d0745e8aae:/src# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31715
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
not sure if this affects things, but running from a Mac M1 using rancher desktop for the docker stuff.
f
maybe set 
ulimit -l unlimited
(for "max locked memory") - I think that LMDB uses mmap to map the on-disk files into memory. Unclear if "max locked memory" is related to mmap usage but easy enough to relax that limit.
c
Iā€™ll try it šŸ™‚
ahemā€¦ operation not permittedā€¦ so I suspect this could be a perm issue where I need to provide more privs to running the container perhaps.
OK, nope, adding all capabilities and changing the -l ulimit to unlimited had unfortunately no effect..
wouldā€™ve been nice if it didnā€™t have to create the lmdb store at all when using 
--no-local-cache
.. likely a technical reason as to why not though.. @witty-crayon-22786?
f
Where is the container running? "rancher" sounds like a locally-installed Kubernetes?
c
no, ā€œrancher desktopā€ is just like ā€œdocker desktopā€ but from rancher with some builtin features for setting up a k8s cluster (not using that).. so itā€™s a regular docker client/server setup on a my machine.
BUT, Iā€™m not sure how the docker setup is configured, compared to that you get with docker desktopā€¦
hmmā€¦ or this is an issue with running an amd64 image on the M1 rather than an arm64 one..
but I donā€™t get it, really, as thereā€™s qemu virtualization going onā€¦ oh well
w
Does this problem still occur if you mount the cache on a separate volume? I vaguely recall playing around with that when I ran into this type of problem way back
r
hmmā€¦ or this is an issue with running an amd64 image on the M1 rather than an arm64 one..
This I have seen happening with my colleagues. It gets stuck on ā€œstarting pantsdā€ for long time and then times out Although I didnā€™t verify if itā€™s the difference in the architecture which is creating this issue.
c
Does this problem still occur if you mount the cache on a separate volume?
Tried to point the lmdb_store to other paths, including a volume mounted one, no avail šŸ˜•
šŸ˜• 1
f
how much memory is available/allocated to the container?
w
itā€™s possible that adjusting the 
local_store_*_max_size_bytes
settings lower (particularly https://www.pantsbuild.org/docs/reference-global#local_store_files_max_size_bytes) will help.
šŸ’Æ 1
ā˜ļø 1
LMDB will mmap the entire store, and that can hit different virtual memory limits in different environments.
wouldā€™ve been nice if it didnā€™t have to create the lmdb store at all when using 
--no-local-cache
.. likely a technical reason as to why not though.. @witty-crayon-22786?
even when the cache is disable, the store is used to snapshot files so that we can then hold 
Digests
for them
āœ… 1
c
how much memory is available/allocated to the container?
Copy code
CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O     BLOCK I/O   PIDS
d3bfea806e90   crazy_davinci   0.00%     9.855MiB / 7.752GiB   0.12%     806B / 0B   0B / 0B     2
itā€™s possible that adjusting the 
local_store_*_max_size_bytes
settings lower will help
Youā€™re an absolute ace, @witty-crayon-22786! šŸ™Œ ā¤ļø For posterity, hereā€™s my settings:
Copy code
export PANTS_LOCAL_STORE_PROCESSES_MAX_SIZE_BYTES=1600000000 PANTS_LOCAL_STORE_FILES_MAX_SIZE_BYTES=2560000000 PANTS_LOCAL_STORE_DIRECTORIES_MAX_SIZE_BYTES=1600000000
export PANTS_WATCH_FILESYSTEM=false PANTS_LOCAL_CACHE=false PANTS_PANTSD=false
That is I dropped the 16GB ones to 1.6GB and the 256GB down to 2.56GB or there abouts..
ā¤ļø 1
w
LMDBā€™s time is definitely limited. it has served us ā€¦ ā€œwellā€, but has three of four different annoying issues like this.
šŸ‘ 1
c
Thanks also to Tom, Shantanu, SJ and David for suggestions and questions šŸ‘
w
glad that worked. note that you might also want to adjust the shard count to prevent limiting the max size of files
šŸ™ 1
h
LMDBā€™s time is definitely limited.
What comes next?
c
might also want to adjust the shard count
thereā€™s a lot of tweakability (for better and worse) here - come to think of ā€œprofilesā€, like slim/local/ci/custom/ā€¦ šŸ˜‰
w
What comes next?
unclear. sqlite, rocksdb, who knows
6 Views
Open in Slack
PreviousNext
Powered by