I keep seeing `MaxFilesWatch` in an irregular fashion In the Pants #general

I keep seeing `MaxFilesWatch` in an irregular fash...

hundreds-lion-13128

11/29/2024, 6:36 PM

I keep seeing

MaxFilesWatch

in an irregular fashion. In the meanwhile, I have added

inotify-info

to my CI image and let it capture inotify info on failure. This is what it reports:

Copy code

INotify Limits:
max_queued_events    16,384
max_user_instances   128
max_user_watches     8,192

Total inotify Watches:   450
Total inotify Instances: 2

So even though the watch count is well below the limits, I get this error. This happens every 1/20 builds or so on CI.

hundreds-lion-13128

11/29/2024, 6:38 PM

Copy code

IntrinsicError: Snapshot failed: Failed to digest inputs: Throw { val: Failed to watch filesystem for `/x.py`: Error { kind: MaxFilesWatch, paths: ["/x.py"] }, python_traceback: "Traceback (no traceback):\n  <pants native internals>\nException: Failed to watch filesystem for `/x.py`: Error { kind: MaxFilesWatch, paths: [\"/x.py\"] }", engine_traceback: [FailureFrame { name: "digest_file", desc: Some("Fingerprinting: x.py") }] }

hundreds-lion-13128

11/29/2024, 6:40 PM

happy-kitchen-89482

12/02/2024, 1:17 PM

Hmm, that is mysterious. The

Total inotify Watches:   450

implies that Pants is not consuming more watches than it should be. It sounds like that is roughly the number of source files in your repo?

hundreds-lion-13128

12/12/2024, 6:29 PM

Jup, it's about that number.

hundreds-lion-13128

12/12/2024, 6:30 PM

I wonder, could it be that the host kernel has a limit, and that Docker containers individually report that number, but that in reality their number of available watches is lower because of other processes watching?

happy-kitchen-89482

12/12/2024, 9:49 PM

Yes, that is likely it. The # of watches is a kernel-level value.

4 Views

Open in Slack

Previous Next