https://pantsbuild.org/ logo
Powered by
c

cold-soccer-63228

05/25/2022, 5:46 PM
I'm encountering a "No space left on device" error while watching files error on my Gitlab CI Runners. The docs suggest to adjust the limit on file watches, but that feels suboptimal for CI machines. Is there a known alternative?
Error Message
Copy code
Exception: Snapshot failed: Failed to digest inputs: Throw { val: Failed to watch filesystem for `/builds/gitlab-group-name/gitlab-project-name/path/to/BUILD`: Error { kind: Io(Os { code: 28, kind: StorageFull, message: "No space left on device" }), paths: [] }
On Linux, this can be caused by a `max_user_watches` setting that is lower than the number of files and directories in your repository (yours is set to 12288). Please see <https://www.pantsbuild.org/docs/troubleshooting#no-space-left-on-device-error-while-watching-files> for more information., python_traceback: "Traceback (no traceback):\n  <pants native internals>\nException: Failed to watch filesystem for `/builds/gitlab-group-name/gitlab-project-name/path/to/BUILD`: Error { kind: Io(Os { code: 28, kind: StorageFull, message: \"No space left on device\" }), paths: [] }\n\nOn Linux, this can be caused by a `max_user_watches` setting that is lower than the number of files and directories in your repository (yours is set to 12288). Please see <https://www.pantsbuild.org/docs/troubleshooting#no-space-left-on-device-error-while-watching-files> for more information.", engine_traceback: ["digest_file"] }
w

witty-crayon-22786

05/25/2022, 5:51 PM
c

cold-soccer-63228

05/25/2022, 5:53 PM
Yeah, I just came from that link. It seems suboptimal to run the following command for CI machines, because these runners are new pods that are constantly spun up.
Copy code
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
(I'm not sure if there's a way to tell our Gitlab pods to run this every time they start up)
w

witty-crayon-22786

05/25/2022, 5:53 PM
if it’s a docker image, you can do it once in the image
i’m adding a note to the FAQ entry, but: a workaround is to disable 
pantsd
and file watching… but that will be much slower if you run multiple commands in CI, as each will need to warm up
i.e., set 
--no-pantsd --no-watch-filesystem
c

cold-soccer-63228

05/25/2022, 5:56 PM
I see. If I'm only running one command, e.g. 
./pants --no-pantsd --no-watch-filesystem test <path-to-directory>::
, there should be no performance hit, right?
w

witty-crayon-22786

05/25/2022, 5:56 PM
correct
c

cold-soccer-63228

05/25/2022, 5:57 PM
Great. I think it should be fine then! Thanks 🙂
w

witty-crayon-22786

05/25/2022, 5:58 PM
and can put those in 
pants.toml
as:
Copy code
[GLOBAL]
# TODO: We disable file watching because these machines have a
# low `fs.inotify.max_user_watches` value. If we start
# running multiple commands, we should fix this.
pantsd = false
watch_filesystem = false
c

cold-soccer-63228

05/25/2022, 5:59 PM
Okay, awesome. For CI specifically, I can create a 
pants.ci.toml
, and paste that above snippet in there, and it should magically work, right?
w

witty-crayon-22786

05/25/2022, 6:00 PM
you need to tell 
pants
to actually look at that file, but yes… one way to do that is by setting 
PANTS_CONFIG_FILES=pants.ci.toml
, which will add 
pants.ci.toml
to the 
--config-files
value
(so both 
pants.toml
and 
pants.ci.toml
will be loaded, in that order)
c

cold-soccer-63228

05/25/2022, 6:03 PM
On CI, I could then run the following?
Copy code
PANTS_CONFIG_FILES=pants.ci.toml ./pants test <path-to-directory>:::
w

witty-crayon-22786

05/25/2022, 6:04 PM
yes. but you could also just 
./pants --config-files=pants.ci.toml test <path-to-directory>::
in that case. the env var is only useful if you want to ensure that multiple commands are covered
c

cold-soccer-63228

05/25/2022, 6:10 PM
Sounds great, thanks Stu!
p

plain-belgium-20952

09/22/2023, 4:07 PM
Hi I'm also having this issue and would like to continue the conversation in this thread. If this is still failing after disabling pantsd and watch_filesystem, do we have to look into increasing the RAM/CPU/disk space of our GitHub hosted runners? Or look into a self hosted runner? Just wondering if there are any other alternatives.
w

witty-crayon-22786

09/22/2023, 4:19 PM
if 
watch_filesystem
is disabled and you’re still seeing it, can you file an issue with some 
-ldebug
output?
p

plain-belgium-20952

09/22/2023, 6:00 PM
👍 1
7 Views
Powered by