In my opinion, community plugins without an official, stable plugin API could be a recipe for some mild disaster, which is why I think it would be good to start stabilizing the plugin API piece by piece. When the plugins are all built into mainline, they benefit from all of the upgrades along the way, so they don't really fall out of sync. I'm generally on your side of this fence though, as I've appreciated over the years the various tools that come with a broad, sanctioned, plugin community. I also find it a lot faster to create plugins in a separate repo, and potentially mainline them down the road. There are some other chats about this subject in the Slack history which do a better job than I can about detailing the pros/cons

I understand the concern about API stability! So far, I've certainly seen my fair share about hiccups - including the interesting incompatibility between the shell-backend and my own OCI-backend in 2.16. But it's been fairly trivial once debugging is done; and the guides have been great. I'd not call it a disaster, not more than any other plugin system... things tend to break no matter what. Maintaining support for multiple versions is a bit extra work for the maintainers, but it's optional - I don't think there should be any requirement for any plugin to support more than stable. And having them in external repos and using a Github action to pull in and test them from core to find out where things break is possible. And for what it's worth; I think the only way to get a good stable plugin API is by actually dogfooding, and that means actually pushing for experimental/clearly stubbed backends to be built as plugins. The process shouldn't be stable API -> plugins, it should be plugins guiding the API stabilization. Otherwise, how do we find the right APIs if they're not being tested from the perspective of an actual plugin?

So, there's several facets to this. The TL;DR is I think we might get to a place where we have more better out-of-repo plugins. It's something I want as a maintainer because one backend can hold up the entire release of Pants. Some loose thoughts: • It's easier to just add another plugin to pants repo. Easier to copy the code from your neighbor, leverage existing code, leverage all the test harnesses and config, leverage the tools, etc... • As SJ touched on, there's API stability. As you say, it isn't Herculean, but it IS extra work for devs and CI • Having plugins baked in could be seen as friendlier to users, they don't have to go hunting and they don't have match versions • We're able to generate and publish quality docs for every susbsytem/target/ backend with it all in repo • Publishing pants is already complicated, spreading that out over multiple plugins could be even more difficult I think we'll get there. But doing so isn't free. It takes resources. And a lot. My own journey has me getting Pants' release streamlined so at the last bullet point isn't as much of a deal

Re: Dogfooding We've kinda been dogfooding so far, but I find the stable API is not just about what's currently in use - because those might bloat what we want to support. When it comes to APIs, the thinner the better for me. I already have the issue where each time I write a plugin, I wonder which of several similar APIs I should be using. Just becomes too much of a mental burden for my small primate brain unfortunately. So some sort of stats on the existing plugins where it's like "Here are the most used APIs - so let's start there" There's also a bit of an onion effect, where we know that some "core" APIs have been stable and unchanged for a long time, while others have had a bit more churn recently.

Agreed on most points. Re: docs - see my thread a few posts up. And https://tgolsson.github.io/awesome-pants/plugins/, which I hacked together during the weekend... I think discoverability isn't necessarily about being built-in, but gathering things up, and having a community that helps propagate itself. There's one extra step to add a plugin from 3rd-party. Is having a list like that really that much worse than somewhere in the long list of docs and languages and backends on Pantsbuild.org? My one strong disagreement is about publishing Pants getting harder - it would get easier? I think? In my world Pants is still only responsible for itself. I'm the maintainer of plugin "foo", so I'm publishing that myself, using some versioning scheme I prefer. I don't think Pants should spread itself over multiple repos or plugins; only that the default for new things should be to put them in separate repos and PyPi. And maybe over time that model will work so well that smaller, more niche, or less stable things get moved out into the community - is the Pants Scala (as an example, no hate!) community big enough for it to be a sanctioned backend? Would it be better off maintained by those who care in a separate repo? etc. Definitely agree on the overload set SJ; each time I add a new goal I have to learn a new idiom 😂 Maybe having some form of roadmap/goal for which things are stable would be helpful... I think in terms of weirdness, REPL ranks highest, and run probably feels the best. Package and publish have weird interactions that aren't expected, etc. And that's just goals, then it's all the utilities, system binaries, etc. It's definitely a hard nut. And some of them are probably unexpectedly used as part of the public API, even now, even if they're not meant like that.

Re: easier publishing. Right now pants "owns" the plugins baked in. Unless someone steps up to own those (and keep the quality up, and the lights on, etc...) We still own them. So it's N more releases

not adding anything to this right now, more than that this is a great discussion and healthy to have so 💯 to everyone involved.

Re: easier publishing. Right now pants "owns" the plugins baked in. Unless someone steps up to own those (and keep the quality up, and the lights on, etc...) We still own them. So it's N more releases

Oh yeah; I'm not making the argument that they should be moved unless there is a good reason and a viable community to keep them going on their own. So whatever is there stays there, released with Pants, until it can be someone else's cross to bear. But if someone comes along and wants to support Erlang, instead of saying "here's how to PR" we say "here's how to make a plugin - and here's the list and communities for other plugins".

Erlang 😍

Well if we were good stewards I think all of the backends would be plugins, so it'd be "just copy XYZ". But I think we're in agreement, for sure. It just boils down to time and energy 🙂

Just a thought that this feels like a good thread to be moved into a GH Discussion

Thanks for bringing this up @gorgeous-winter-99296! Would you be able to create a GitHub Discussion and link to this thread there? It's an important topic, and it would be a shame if it got lost in the Slack scrollback

Will do!

I added a couple thoughts :)