https://pantsbuild.org/ logo
Join Slack
Powered by
I'm encountering a <https://www.pantsbuild.org/doc...
# general
c
I'm encountering a "No space left on device" error while watching files error on my Gitlab CI Runners. The docs suggest to adjust the limit on file watches, but that feels suboptimal for CI machines. Is there a known alternative?
Error Message
Copy code
Exception: Snapshot failed: Failed to digest inputs: Throw { val: Failed to watch filesystem for `/builds/gitlab-group-name/gitlab-project-name/path/to/BUILD`: Error { kind: Io(Os { code: 28, kind: StorageFull, message: "No space left on device" }), paths: [] }
On Linux, this can be caused by a `max_user_watches` setting that is lower than the number of files and directories in your repository (yours is set to 12288). Please see <https://www.pantsbuild.org/docs/troubleshooting#no-space-left-on-device-error-while-watching-files> for more information., python_traceback: "Traceback (no traceback):\n  <pants native internals>\nException: Failed to watch filesystem for `/builds/gitlab-group-name/gitlab-project-name/path/to/BUILD`: Error { kind: Io(Os { code: 28, kind: StorageFull, message: \"No space left on device\" }), paths: [] }\n\nOn Linux, this can be caused by a `max_user_watches` setting that is lower than the number of files and directories in your repository (yours is set to 12288). Please see <https://www.pantsbuild.org/docs/troubleshooting#no-space-left-on-device-error-while-watching-files> for more information.", engine_traceback: ["digest_file"] }
w
c
Yeah, I just came from that link. It seems suboptimal to run the following command for CI machines, because these runners are new pods that are constantly spun up.
Copy code
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
(I'm not sure if there's a way to tell our Gitlab pods to run this every time they start up)
w
if it’s a docker image, you can do it once in the image
i’m adding a note to the FAQ entry, but: a workaround is to disable 
pantsd
and file watching… but that will be much slower if you run multiple commands in CI, as each will need to warm up
i.e., set 
--no-pantsd --no-watch-filesystem
c
I see. If I'm only running one command, e.g. 
./pants --no-pantsd --no-watch-filesystem test <path-to-directory>::
, there should be no performance hit, right?
w
correct
c
Great. I think it should be fine then! Thanks 🙂
w
and can put those in 
pants.toml
as:
Copy code
[GLOBAL]
# TODO: We disable file watching because these machines have a
# low `fs.inotify.max_user_watches` value. If we start
# running multiple commands, we should fix this.
pantsd = false
watch_filesystem = false
c
Okay, awesome. For CI specifically, I can create a 
pants.ci.toml
, and paste that above snippet in there, and it should magically work, right?
w
you need to tell 
pants
to actually look at that file, but yes… one way to do that is by setting 
PANTS_CONFIG_FILES=pants.ci.toml
, which will add 
pants.ci.toml
to the 
--config-files
value
(so both 
pants.toml
and 
pants.ci.toml
will be loaded, in that order)
c
On CI, I could then run the following?
Copy code
PANTS_CONFIG_FILES=pants.ci.toml ./pants test <path-to-directory>:::
w
yes. but you could also just 
./pants --config-files=pants.ci.toml test <path-to-directory>::
in that case. the env var is only useful if you want to ensure that multiple commands are covered
c
Sounds great, thanks Stu!
p
Hi I'm also having this issue and would like to continue the conversation in this thread. If this is still failing after disabling pantsd and watch_filesystem, do we have to look into increasing the RAM/CPU/disk space of our GitHub hosted runners? Or look into a self hosted runner? Just wondering if there are any other alternatives.
w
if 
watch_filesystem
is disabled and you’re still seeing it, can you file an issue with some 
-ldebug
output?
p
👍 1
9 Views
Open in Slack
PreviousNext
Powered by