https://pantsbuild.org/ logo
c

curved-television-6568

09/13/2022, 2:57 PM
Issue with running pants in a ubuntu container image. 🧵
1
Hi, Trying to run
./pants version
from within a ubuntu docker image, it keeps failing for me with:
Copy code
[...]
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 507, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  Exception: Error making env for store at "/root/.cache/pants/lmdb_store/files/0": Cannot allocate memory

NoneType: None
I’m invoking it thus:
Copy code
./pants --no-pantsd --no-watch-filesystem --no-local-cache version
but even if I select another location for the lmdb store it gives the same kind of error.
f

fast-nail-55400

09/13/2022, 2:59 PM
What are the ulimit's?
(since
Cannot allocate memory
)
c

curved-television-6568

09/13/2022, 2:59 PM
Copy code
root@62d0745e8aae:/src# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31715
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
not sure if this affects things, but running from a Mac M1 using rancher desktop for the docker stuff.
f

fast-nail-55400

09/13/2022, 3:01 PM
maybe set
ulimit -l unlimited
(for "max locked memory") - I think that LMDB uses mmap to map the on-disk files into memory. Unclear if "max locked memory" is related to mmap usage but easy enough to relax that limit.
c

curved-television-6568

09/13/2022, 3:01 PM
I’ll try it 🙂
ahem… operation not permitted… so I suspect this could be a perm issue where I need to provide more privs to running the container perhaps.
OK, nope, adding all capabilities and changing the -l ulimit to unlimited had unfortunately no effect..
would’ve been nice if it didn’t have to create the lmdb store at all when using
--no-local-cache
.. likely a technical reason as to why not though.. @witty-crayon-22786?
f

fast-nail-55400

09/13/2022, 3:20 PM
Where is the container running? "rancher" sounds like a locally-installed Kubernetes?
c

curved-television-6568

09/13/2022, 3:29 PM
no, “rancher desktop” is just like “docker desktop” but from rancher with some builtin features for setting up a k8s cluster (not using that).. so it’s a regular docker client/server setup on a my machine.
BUT, I’m not sure how the docker setup is configured, compared to that you get with docker desktop…
hmm… or this is an issue with running an amd64 image on the M1 rather than an arm64 one..
but I don’t get it, really, as there’s qemu virtualization going on… oh well
w

wide-midnight-78598

09/13/2022, 3:59 PM
Does this problem still occur if you mount the cache on a separate volume? I vaguely recall playing around with that when I ran into this type of problem way back
r

refined-addition-53644

09/13/2022, 4:30 PM
hmm… or this is an issue with running an amd64 image on the M1 rather than an arm64 one..
This I have seen happening with my colleagues. It gets stuck on “starting pantsd” for long time and then times out Although I didn’t verify if it’s the difference in the architecture which is creating this issue.
c

curved-television-6568

09/13/2022, 4:31 PM
Does this problem still occur if you mount the cache on a separate volume?
Tried to point the lmdb_store to other paths, including a volume mounted one, no avail 😕
😕 1
f

freezing-lamp-12123

09/13/2022, 4:34 PM
how much memory is available/allocated to the container?
w

witty-crayon-22786

09/13/2022, 4:35 PM
it’s possible that adjusting the
local_store_*_max_size_bytes
settings lower (particularly https://www.pantsbuild.org/docs/reference-global#local_store_files_max_size_bytes) will help.
💯 1
☝️ 1
LMDB will mmap the entire store, and that can hit different virtual memory limits in different environments.
would’ve been nice if it didn’t have to create the lmdb store at all when using
--no-local-cache
.. likely a technical reason as to why not though.. @witty-crayon-22786?
even when the cache is disable, the store is used to snapshot files so that we can then hold
Digests
for them
1
c

curved-television-6568

09/13/2022, 6:05 PM
how much memory is available/allocated to the container?
Copy code
CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O     BLOCK I/O   PIDS
d3bfea806e90   crazy_davinci   0.00%     9.855MiB / 7.752GiB   0.12%     806B / 0B   0B / 0B     2
it’s possible that adjusting the
local_store_*_max_size_bytes
settings lower will help
You’re an absolute ace, @witty-crayon-22786! 🙌 ❤️ For posterity, here’s my settings:
Copy code
export PANTS_LOCAL_STORE_PROCESSES_MAX_SIZE_BYTES=1600000000 PANTS_LOCAL_STORE_FILES_MAX_SIZE_BYTES=2560000000 PANTS_LOCAL_STORE_DIRECTORIES_MAX_SIZE_BYTES=1600000000
export PANTS_WATCH_FILESYSTEM=false PANTS_LOCAL_CACHE=false PANTS_PANTSD=false
That is I dropped the 16GB ones to 1.6GB and the 256GB down to 2.56GB or there abouts..
❤️ 1
w

witty-crayon-22786

09/13/2022, 6:21 PM
LMDB’s time is definitely limited. it has served us … “well”, but has three of four different annoying issues like this.
👍 1
c

curved-television-6568

09/13/2022, 6:23 PM
Thanks also to Tom, Shantanu, SJ and David for suggestions and questions 👍
w

witty-crayon-22786

09/13/2022, 6:23 PM
glad that worked. note that you might also want to adjust the shard count to prevent limiting the max size of files
🙏 1
h

hundreds-father-404

09/13/2022, 6:23 PM
LMDB’s time is definitely limited.
What comes next?
c

curved-television-6568

09/13/2022, 6:24 PM
might also want to adjust the shard count
there’s a lot of tweakability (for better and worse) here - come to think of “profiles”, like slim/local/ci/custom/… 😉
w

witty-crayon-22786

09/13/2022, 6:24 PM
What comes next?
unclear. sqlite, rocksdb, who knows
6 Views