When I check out a different branch that has e.g. a different plugin configuration, the pantsd server does not seem to restart and I see errors like

21:00:00.00 [ERROR] Invalid table name [new-subsystem-config] in pants.toml
21:00:00.00 [ERROR] Invalid config entries detected. See log for details on which entries to update or remove.

Similarly, when my checkout removes targets in BUILD files, the server doesn't seem to notice and I get warnings like "no matching files for glob '...' in 'old_target'".

Huh, it should restart due to changes to pants.toml

Yes, this happens on mine and others (all Macs). A related issue that I see is that when changing branches, pants warns about targets being 'empty', even though those targets don't exist in the new branch.

17:00:00.00 [WARN] Unmatched glob from some/file.py's `source` field: "some/file.py"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/troubleshooting#pants-cannot-find-a-file-in-your-project>.

I regularly see the second issue, but haven't seen the issue with

pants.toml

before. Can you construct a simple reproduction repo?

So it's not happening reliably, but it seems to happen when someone does a big checkout (many files) including an addition to a plugin and pants.toml update. So could it be that the file watcher queue overflows and discards some events which makes pantsd skip the restart?

This is on MacOS, right?

Sadly, this happens quite regularly in our system. We have ~100 MacOS users and >10 have reported this issue to me.

If I read the code correctly, the error in your original message happens on your next pants invocation, if pantsd lives over a branch switch and pants.toml has changed?

Hmm, the backend loader happens early, before full options parsing, maybe that is related

Interesting, I'll dig into it. One side-effect of it not being a daemon option is it doesn't end up as part of the fingerprint, which is one reason to restart pantsd. But maybe the backend loader handles that some other way.

Also; do you have any Linux users that are never seeing this

We only use Linux in CI and some edge cases. But because we don't do frequent branch-changes and other large-scale file operations there after booting up pants, we haven't seen it there.

Just double-checking that you don’t have any custom

pants_ignore

in your pants.toml? Or anything funky in

.gitignore

?

No, only a

["!/sandbox/"]

entry in pants_ignore, and

.pants.d/
/sandbox/
__pycache__/
*.py[cod]
....

in gitignore

I figured, but just wanted to close out that unlikely corner

Yes, what makes it more likely seems to be doing large checkouts, so a lot of files change at once. I haven't been able to create a super reliable repro, but I started with just creating 20k bogus Python files, committing them to a branch. Then create another 20k of other bogus files, commit them. Then switch between those branches.