https://pantsbuild.org/ logo
Join Slack
Powered by
I have an issue where the Pants daemon appears to ...
# general
a
I have an issue where the Pants daemon appears to restart almost every time I want to run any test file. Is there a way to debug exactly why it's restarting? I'm not changing any global options between the runs, and am rerunning a command such as 
pants test --debug /path/to/file -- -s.
c
unless it gets killed by oom or something like that.. it usually logs in 
.pants.d/pants.log
why it restarts, I think. (or if it died due to some error.)
👍 1
h
Another possible cause is if you're modifying an in-repo plugin
But if you're not even editing files between two test runs, then it could be OOM
a
There's no indication of swapping and ~25Gb of free memory, so OOM should be out of the question unless something really weird is happening. In the log I'm seeing 
_09:41:46.50_ [ERROR] saw filesystem changes covered by invalidation globs: content changed
followed by a list of digests. Is it possible for me to map this digest back to the original filename?
c
usually the filenames in question are logged... this being an error message indicates something unexpected happened, I think...
@witty-crayon-22786 would know more perhaps 😉
a
It's happening across multiple developer machines, if that helps, so it's not isolated to my local system.
Unfortunately there's not much else in the log. Right after this error another error is logged:
Copy code
09:50:52.99 [ERROR] service failure for <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f6f861c9e20>.
h
Sounds like the pantsd_invalidation_globs are being triggered
Are you setting that in config somewhere?
Note that pants also invalidates if code on the sys.path changes.
On pants's sys.path that is
a
Hmm. I'm not setting the invalidation globs explicitly at least. What constitutes a change of Pant's sys.path? What are some common operations that cause this?
Picking this up again. I'm able to consistently reproduce this when I change a given test file, e.g. adding or removing an empty test logs:
Copy code
16:04:03.15 [INFO] notify invalidation: cleared 1 and dirtied 13 nodes for: {"/path/to/module/tests/integration/test_search.py"}
16:04:03.15 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"/path/to/module/tests/integration/test_search.py"}
16:04:03.17 [INFO] notify invalidation: cleared 2 and dirtied 107 nodes for: {{"/path/to/module/tests/integration"}, {"/path/to/module/tests/integration/test_search.py"}}
16:04:03.20 [ERROR] saw filesystem changes covered by invalidation globs: content changed (Digest('f48528327e3da7bf8a3c73d1b8dd9383091c993ec007b1fc0953a1fedd7660fc', 581) fs Digest('450e4b274d1fb97e14e33fe9596df3fb8a432501829b81ca2a55d49eaa951d7e', 581)). terminating the daemon.
16:04:03.87 [ERROR] service failure for <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f0e42eabee0>.
16:04:03.87 [INFO] Waiting for ongoing runs to complete before exiting...
16:04:03.87 [INFO] Server exiting with Ok(())
16:04:03.87 [INFO] Waiting for Sessions to complete before exiting...
16:04:03.87 [INFO] Waiting for shutdown of: ["scheduler_service_session", "store_gc_service_session", "pants_run_2024_03_11_16_03_51_895_a92c50e7e1074628a428118edf718a90"]
16:04:03.87 [INFO] Shutdown completed: "scheduler_service_session"
16:04:03.87 [INFO] Shutdown completed: "store_gc_service_session"
16:04:03.87 [INFO] Shutdown completed: "pants_run_2024_03_11_16_03_51_895_a92c50e7e1074628a428118edf718a90"
16:04:03.88 [INFO] Exiting pantsd
16:04:03.89 [WARN] File watcher exiting with: The watcher was shut down.
I don't have any invalidation globs explicitly set. I'm trying to narrow it down further, but any help is greatly appreciated.
w
The list of files listed above the error as "notify invalidation" files is the relevant bit.
If any of those files match paths configured as invalidation global for pantsd, then pantsd will restart
a
Yes, but is there any way I could have accidentally set any invalidation globs? Because I'm not setting in the command, nor is it configured in 
pants.toml
. That's the part that's really confusing to me. Maybe there's a way to list all invalidation globs?
w
There are defaults. To list them, can use 
pants help global
, iirc.
a
Hmm, using 
ldebug
makes pants log debug info to pants.log as well, that's neat. This produces an output line:
Copy code
16:11:53.81 [DEBUG] watching invalidation patterns: ('libs/...', ..., ..., ..., ...)
These would also be the relevant bits, right?
w
Yep!
a
That lists all of my source roots though. Isn't that a bit weird?
w
Did you add all of your sourceroots to Pants' pythonpath?
That has the effect of putting them on the pythonpath of Pants itself... i.e., to treat them as plugins
a
At the risk of sounding very dumb ... I don't think I have? I've declared all my source roots in 
[source].root_patterns
, but not anywhere else.
w
a
No, we haven't modified that at all.
w
are you able to share your config in a DM?
a
Yes, I can strip it of all "secrets" 🙂 Hang on
Sent!
3 Views
Open in Slack
PreviousNext
Powered by