#20193 Proposal: run the full test suite on all platforms, more often New discussion created by huonw Currently, CI runs the full Pants test suite on Linux x86-64, via Github's free runners, but a only limited subset of them (marked

platform_specific_behavior

"PSB") on the other three supported platforms, if at all. This means that problems can slip through when we don't realise something is platform specific, e.g. #20184. Summary of what CI does, as best I can tell: ^Rust tests and wheel-building only runs when required. That is, there's some platforms that are completely untested, and others that only ever have limited testing. Proposal: Running the full test suite on the hosted machines more often, somehow. For instance: 1. Running everything on all platforms, for every PR CI run: this almost certainly takes too much CI resources. 2. Use a merge queue that runs the full test suite on all platforms, while normal PR CI runs a reduced set (i.e. fast/low-resource-usage feedback for iteration, but follow the "not rocket science" rule for code to actually land https://graydon2.dreamwidth.org/1597.html ). Potentially normal PR CI could be reduced further since it's no longer the main gate, e.g. only run on the free GitHub-hosted runners, and use none of the self-hosted runners. 3. A semi-regular build of

main

and

2.*.x

branches, e.g. nightly, weekly, that the MOTW checks 4. A full test run either just-before or just-after a release, as part of the release process, again checked by the MOTW. 5. Something else? I'm personally inclined towards 2, potentially using Github's recent-ish support for merge queues: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/merging-a-pull-request-with-a-merge-queue What do you think? pantsbuild/pants