I've been avoiding writing tests for some plugins. The testing feels like a different level of abstraction and I question the value of tests in some cases. So, here's an example formatter plugin that runs some in-repo code as a Pex in a very simple rule. https://github.com/StackStorm/st2/compare/pants-plugins-schemas So, what would you test here? What value would the different kinds of tests listed in the testing docs provide for this plugin?

Some errors are not detected until runtime, for example rule graph errors.

Take it from someone with several plugins and no tests. They all break eventually. Even a smoke test goes a long way

I arguably overtest my internal plugins, even though some of them only I use. It's just a habit I've gotten into, and most of the tests are just sanities (does it run, is the result roughly correct) However, the tests can be useful to verify against regressions if you change your Pants version. I think that would be first and foremost. That's burned me a couple of times. Otherwise, if the plugin is simple enough and there is some level of constant runtime checking (e.g running the plugin in CI against your code or something), then the value of tests for small/simple tools is diminished. However, if you're pulling in optional config files, lots of options, fields, etc - then yeah. definitely more useful to test.

Oh. Pants upgrades. Yeah that's a really good point.