https://pantsbuild.org/ logo
#development
Title
# development
c

careful-address-89803

08/01/2022, 5:04 AM
hey folks, I was looking at an older project of mine that used Tox and was appreciating how easy it was to "add" a "plugin":
Copy code
[testenv:radon]
deps = radon
commands = 
    radon cc -s --total-average --no-assert -nb src/
I thought it would be great if it were that easy to create a linter in Pants. So I hacked together a prototype here, and it looks like it "works" and all you need for a linter is:
Copy code
def rules():
    radoncc = make_linter(
        "radoncc",
        "radon",
        "radon==5.1.0",
        ["cc", "-s", "--total-average", "--no-assert", "-n"],
    )

    return radoncc.rules()
2 questions: 1. is this a horrible hack that will explode? (I know it's a horrible hack that needs a lot of cleanup) 2. is this (or these sorts of helpers) something we'd like to include in Pants?
🤯 1
👀 2
w

wide-midnight-78598

08/01/2022, 12:37 PM
I've gotta assume for a trivial base-case, something like this could exist - and then any customization can result in what we do today. Having made a lot of small plugins, the code you have is pretty reflective of the copy/pasting I do too As you've split out here, most of the "thought" when building some of the Python plugins is around the subsystem and args. You might also need to consider if the tool needs transitive deps? But maybe that's more for
check
goals. Maybe it's worth eye-balling some of the existing linters/formatters, to see whether this would be a reasonable replacement? Specifically I'm thinking about things like config files, ensuring help still works, could they be lockfile'd, export'ed, how would unit testing of the "made" linter work, etc?
Also, if not this exactly, I think there is room for some sort of code generator, or template project, or cookie cutter, or something that does this - give it a name, commands, etc - and have it just generate a reasonable plugin of that form (check, lint, format, etc) I tried to start something like that here (https://github.com/sureshjoshi/pants-plugin-template/tree/1-empty-scaffold) - but was pulled away for other Pants stuff.
b

bitter-ability-32190

08/01/2022, 2:32 PM
Yeah. I'll say, in general, my eventual long-term goal is for Pants to be even more declarative and simplistic. The hard part is the je ne sais quoi of simplicity v customizability
2
As SJ points out, the balance between subsystems and args and user-customizability makes this hard. You and I might think we want a specific set of args, but what about your entire engineering org? What about when you need a verbosity flag? Or some other flag?
h

hundreds-father-404

08/01/2022, 3:10 PM
Cool! We've generally biased towards less DRY until we're confident in it because we've seen enough use cases. But we've seen so many linters now, there's room for better de-duplication The new
@rule_helper
thing helps, that you can factor out rule logic now, even if they need to do
await Get
inside the function
b

bitter-ability-32190

08/01/2022, 3:16 PM
I'd love to see the PR though, even just for demonstration! It's good to diffuse new ideas throughout the community 😁
💯 1
❤️ 3
1
c

careful-address-89803

08/01/2022, 3:19 PM
Hmm, those are all features which I'd need to see if they can be added (I hadn't even thought about looking into help, but it would probably be something like that which would break). I think they should be possible to support mostly automatically. I'm not sure about transitive dependencies, I've never looked into those. As for the balance between standard config and customisability, the guiding principle I use is from mechanical engineering: always provide maintenance hatches.
b

bitter-ability-32190

08/01/2022, 3:22 PM
Pants is flexible enough I think we're able to provide these "shortcuts", knowing the full plugin API is still available for use if it doesn't fit. It's more a matter of maintenance of several APIs and documentation that starts making us cautious
👍 2
h

hundreds-father-404

08/01/2022, 3:26 PM
always provide maintenance hatches.
I like that principle. We try to do this, but haven't used the same phrasing
c

careful-address-89803

08/01/2022, 3:34 PM
in re DRY: My idea for this was less about DRY and more about removing the amount of things someone needs to know to just get off the ground. Many projects will have a few tools that are just "install the things, run the thing", and I think having a simple way to "just get those into Pants" will help with onboarding those projects. I hadn't thought of it as a separate API, although I agree it is.
b

bitter-ability-32190

08/01/2022, 3:55 PM
In that case, I almost wonder if this can be shifted even farther. Make it completely a config opt-in, without any knowledge of "plugins" or "rules" 🤔
w

wide-midnight-78598

08/01/2022, 5:06 PM
Eric made a good point about
rule_helpers
, I wonder if we can create some composable rule_helpers for common workflows, to reduce even more cruft. It doesn't go as far as Daniel or Josh are suggesting (yet), but I like the idea of a lot of the common tools/workflows for linters/formatters are basically just
Subsystems
, and any bootstrapping/custom tweaks can be in the subsystem file as a bootstrapper rule. THAT would be excellent - as today a pretty simple lint plugin can be 6-8 files and 600-700 lines of code (including tests), about 30-40 of which are legitimately "novel". If this could get closer to 3-4 files and 100-200 lines of targeted code, that would be amazing
1
c

careful-address-89803

08/01/2022, 5:35 PM
yeah, that would be helpful. I've got a probably-working function that adds support for
export
to a PythonToolBase. Things like that for other features would be neat
Hey folks, I think I've got enough of a proof of concept that we can decide whether/how this gets included in Pants and what features it needs to have. What's a good format for presenting this?
b

bitter-ability-32190

08/02/2022, 1:07 PM
Usually I post a draft PR and maybe take a screen record if that's helpful 🤷‍♂️
Most of the people here are smart and imaginative, so that helps. Although some examples/use-case go a long way
c

careful-address-89803

08/02/2022, 1:25 PM
sounds good. Where's a good place for these to live in the codebase?
src/python/pants/backend/python/
?
b

bitter-ability-32190

08/02/2022, 3:49 PM
🤷‍♂️ Pick one and we could always discuss in PR 🙂
c

careful-address-89803

08/02/2022, 3:50 PM
Haha sounds good
RFC, a helper to generate linters
👍 1
w

wide-midnight-78598

08/15/2022, 12:23 PM
Doh! I never saw this, I was just waiting for a PR 🤦‍♂️ ... rather, I thought the RFC was a PR that I've been trying to find for 10 minutes Edit: I'm tired and a moron - I kept looking for "metalinter"
b

bitter-ability-32190

08/15/2022, 1:00 PM
I owe this a closer inspection. I think an easy way to make sure it can make forward progress is to find the seams and try and make incremental progress. For instance I think the Python tool base can provide the lockfile stuff and every? tool could easily benefit
b

bitter-ability-32190

08/15/2022, 2:39 PM
The need to declare it, yes. It's be something like
*ClangFormat.lockfile_rules()
in the module's
rules
function I suspect
c

careful-address-89803

08/15/2022, 3:25 PM
yeah, extracting the functionality to introduce gradually makes sense as a path forward. I think "export" and "lockfile" are the actions that would be best to target first I'll give that a look when I get off of work