https://pantsbuild.org/ logo
#general
Title
# general
b

broad-processor-92400

11/22/2022, 8:56 PM
I'm trying to run pants 2.14 within a docker image on a arm mac host, to simulate CI (i.e. where pants is running on Linux), and hit the following error:
Copy code
$ docker run -it --platform linux/amd64 python:3.9 bash -c 'echo -e "[GLOBAL]\npants_version = \"2.14.0\"" > pants.toml && curl -L -O <https://static.pantsbuild.org/setup/pants> && chmod +x ./pants && ./pants version'
... setup ...
New virtual environment successfully created at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39.
20:52:23.81 [INFO] waiting for pantsd to start...
20:52:28.81 [INFO] waiting for pantsd to start...
Traceback (most recent call last):
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 215, in __init__
    self._py_scheduler = native_engine.scheduler_create(
ValueError: Failed to begin watching the filesystem: Function not implemented (os error 38)
20:52:33.83 [INFO] waiting for pantsd to start...
... continues like that ~forever ...
Is there a way to get this to work? _update_: solution: run like
./pants --no-pantsd --no-watch-filesystem ...
and make sure the docker containers have a high enough memory limit
1
b

bitter-ability-32190

11/22/2022, 8:57 PM
h

hundreds-father-404

11/22/2022, 8:58 PM
Very unfortuantely, file watching doesn't work when using Docker on M1s due to Rosetta not properly supporting
inotify
. See last paragraph of: https://www.pantsbuild.org/docs/prerequisites#macos
But there's a neat alternative coming in 2.15 worth considering: Pants gains support for performing individual "builds" like test runs and packaging binaries via Docker. You run
./pants
from your localhost (mac), and Pants will use Docker for you appropriately Any interest in that? What commands are you running?
b

broad-processor-92400

11/22/2022, 9:02 PM
Thanks both. Ah, I think the wheel is fine (it's running a
--platform linux/amd64
image under emulation 🐌), but yeah,
--no-watch-filesystem
seems to resolve it. Thanks.
I'm actually trying to reproduce an issue related to pants itself that we're seeing on CI, so the upcoming environments feature probably doesn't quite solve it (as in, it seems to be a bug with pants-in-Linux, and so I need a Linux system to reproduce, and the easiest one I have access to is docker)
👍 1
Hm, adding
--no-pantsd --no-watch-filesystem
stops the exception, but it now just dies: logs with
-ldebug
Copy code
21:14:01.84 [DEBUG] File handle limit is: 1048576
21:14:04.65 [DEBUG] Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }] for process execution.
21:14:05.74 [DEBUG] Launching 1 roots (poll=false).
21:14:05.88 [DEBUG] computed 1 nodes in 0.137155 seconds. there are 7 total nodes.
21:14:06.19 [DEBUG] File handle limit is: 1048576
Killed
1
h

hundreds-father-404

11/22/2022, 9:31 PM
huh...not sure why that would be. maybe OOM? if you haven't yet, try a simple command like
./pants --version
b

broad-processor-92400

11/22/2022, 9:37 PM
Unfortunately, that's the output from
./pants -ldebug --no-pantsd --no-watch-filesystem version
. Running under
/usr/bin/time -v
shows
Maximum resident set size (kbytes): 2640740
, and watching
top
indeed shows resident memory of the Python process getting to 2.5G, and virtual memory getting to... 480G
that's in a directory that is literally just the pants script and `pants.toml`:
Copy code
[GLOBAL]
pants_version = "2.14.0"
(plus the
.pants.d
that pants creates)
Ah okay, increasing the memory limit of the docker images works, but I've filed https://github.com/pantsbuild/pants/issues/17622 because it'd be nice to use less!
w

witty-crayon-22786

11/22/2022, 11:28 PM
the virtual memory aspect is due to LMDB, which `mmap`s its entire storage. but 2.5G of resident memory is very surprising… i see about 80-140MB outside of docker
👍 1
14 Views