I'm seeing every so often an error

sqlite3.DatabaseError: database disk image is malformed

When running

pants check

on my mid-size Python codebase on CI. I can't ssh into my ssh, sadly. And it does have caching, but shouldn't bee accessed by multiple jobs at the same time.

Just confirming: this is using mypy? Is there any other error logging/context available?

There is very little logging info. This is using Mypy, yes!

Hmm. Very non-obvious! • Pants should be configuring this cache to be resilient to concurrent runs, but sounds like that's not happening here anyway • Pants does some copying/linking between the persistent cache and the per-run cache file actually used when mypy executes: https://github.com/pantsbuild/pants/blob/6c5df42c48ab98a1cb6ae47a7600ce8dfa9f36da/src/python/pants/backend/python/typecheck/mypy/rules.py#L254-L313 Brainstorming ways that the file might be malformed: 1. Mypy is corrupting its own file somehow 2. There's different versions of SQLite in play, e.g. run 1 creates the file with "new SQLite" and run 2 tries to read it with "old SQLite", where the versions/file-formats differ 3. Pants is corrupting the file while initialising the per-run cache from the persistent cache 4. Pants is corrupting the file while saving the persistent cache after a run 5. The CI orchestration is corrupting the persistent cache file between jobs I'm not sure of the best way to narrow down the cause, especially if you can't yet reproduce it reliably/outside CI. Some ideas: • Which version of Pants and mypy is involved? • Add

[mypy].args = ["--verbose", "--show-traceback]

to pants.toml to see if mypy itself can tell us anything more (https://www.pantsbuild.org/prerelease/reference/subsystems/mypy#args) • Do some introspection of the persistent caches, e.g.

sha256sum ~/.cache/pants/named_caches/mypy_cache/*/*/cache.db

at various points of the CI flow and check that the values match, and/or

sqlite ...

invocations to validate the files externally • Temporarily replace the built-in mypy backend with a local plugin (i.e. all the code so we can edit it), and hack in more validation

Wild speculation, but I wonder if there is some shared cause with some of the reports in https://github.com/pantsbuild/pants/issues/18519