#18400 `FAILED tests/python/pants_test/pantsd/pantsd_integration_test.py::TestPantsDaemonIntegration::test_pantsd_invalidation_pants_toml_file` Issue created by kaos Flaky test. Seen this fail frequently for PR runs. One example: https://github.com/pantsbuild/pants/actions/runs/4324001344/jobs/7548328792

=================================== FAILURES ===================================
_____ TestPantsDaemonIntegration.test_pantsd_invalidation_pants_toml_file ______

self = <pants_test.pantsd.pantsd_integration_test.TestPantsDaemonIntegration testMethod=test_pantsd_invalidation_pants_toml_file>

    def test_pantsd_invalidation_pants_toml_file(self):
        # Test tmp_pants_toml (--pants-config-files=$tmp_pants_toml)'s removal
        tmp_pants_toml = os.path.abspath("testprojects/test_pants.toml")
    
        # Create tmp_pants_toml file
        with safe_open(tmp_pants_toml, "w") as f:
            f.write("[DEFAULT]\n")
    
        with self.pantsd_successful_run_context() as ctx:
            ctx.runner([f"--pants-config-files={tmp_pants_toml}", "help"])
            ctx.checker.assert_started()
            time.sleep(10)
    
            # Delete tmp_pants_toml
            os.unlink(tmp_pants_toml)
>           ctx.checker.assert_stopped()

tests/python/pants_test/pantsd/pantsd_integration_test.py:388: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/python/pants_test/pantsd/pantsd_integration_test_base.py:138: in assert_stopped
    for _ in attempts("pantsd should be stopped!"):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

msg = 'pantsd should be stopped!'

    def attempts(
        msg: str,
        *,
        delay: float = 0.5,
        timeout: float = 30,
        backoff: float = 1.2,
    ) -> Iterator[None]:
        """A generator that yields a number of times before failing.
    
        A caller should break out of a loop on the generator in order to succeed.
        """
        count = 0
        deadline = time.time() + timeout
        while time.time() < deadline:
            count += 1
            yield
            time.sleep(delay)
            delay *= backoff
>       raise AssertionError(f"After {count} attempts in {timeout} seconds: {msg}")
E       AssertionError: After 15 attempts in 30 seconds: pantsd should be stopped!

tests/python/pants_test/pantsd/pantsd_integration_test_base.py:55: AssertionError
----------------------------- Captured stdout call -----------------------------

pantsd log is /tmp/pants-sandbox-E38ruW/tmp8sfk9de2/.workdir.pants.d/pantsd/pantsd.log
>>> config: 
{'GLOBAL': {'pantsd': True, 'level': 'info', 'pants_subprocessdir': '/tmp/pants-sandbox-E38ruW/tmp8sfk9de2/.pids', 'backend_packages': ['pants.backend.python', 'pants.backend.python.lint.flake8']}}


running: ./pants --pants-config-files=/tmp/pants-sandbox-E38ruW/testprojects/test_pants.toml help (config={'GLOBAL': {'pantsd': True, 'level': 'info', 'pants_subprocessdir': '/tmp/pants-sandbox-E38ruW/tmp8sfk9de2/.pids', 'backend_packages': ['pants.backend.python', 'pants.backend.python.lint.flake8']}}) (extra_env=None)

completed in 6.701683282852173 seconds
PantsDaemonMonitor: pid is 7907 is_alive=True
PantsDaemonMonitor: pid is 7907 is_alive=True
===============================================================
- BEGIN pants.log ---------------------------------------------
===============================================================
13:53:03.30 [INFO] pantsd 2.16.0.dev7 running with PID: 7907
13:53:03.33 [INFO] handling request: `--no-pantsrc --pants-workdir=/tmp/pants-sandbox-E38ruW/tmp8sfk9de2/.workdir.pants.d --pants-ignore=+['.coverage.*'] --pants-config-files=[] --pants-ignore=+['/conftest.py'] --pants-config-files=/tmp/pants-sandbox-E38ruW/tmp8sfk9de2/.workdir.pants.d/pants.toml --pants-config-files=/tmp/pants-sandbox-E38ruW/testprojects/test_pants.toml help`
13:53:05.50 [INFO] request completed: `--no-pantsrc --pants-workdir=/tmp/pants-sandbox-E38ruW/tmp8sfk9de2/.workdir.pants.d --pants-ignore=+['.coverage.*'] --pants-config-files=[] --pants-ignore=+['/conftest.py'] --pants-config-files=/tmp/pants-sandbox-E38ruW/tmp8sfk9de2/.workdir.pants.d/pants.toml --pants-config-files=/tmp/pants-sandbox-E38ruW/testprojects/test_pants.toml help`
13:53:15.70 [INFO] notify invalidation: cleared 2 and dirtied 2 nodes for: {"testprojects", "testprojects/test_pants.toml"}
13:53:15.70 [INFO] notify invalidation: cleared 2 and dirtied 2 nodes for: {"testprojects", "testprojects/test_pants.toml"}
13:53:15.70 [INFO] Filesystem changed during run: retrying `Snapshot` in 500ms...
===============================================================
- END pants.log -----------------------------------------------
===============================================================
- generated xml file: /tmp/pants-sandbox-E38ruW/tests.python.pants_test.pantsd.pantsd_integration_test.py.pantsd_integration.xml -
=========================== short test summary info ============================
FAILED tests/python/pants_test/pantsd/pantsd_integration_test.py::TestPantsDaemonIntegration::test_pantsd_invalidation_pants_toml_file
============= 1 failed, 24 passed, 2 skipped in [432](<https://github.com/pantsbuild/pants/actions/runs/4324001344/jobs/7548328792#step:11:433>).41s (0:07:12) ==============

pantsbuild/pants