https://pantsbuild.org/ logo
Join Slack
Powered by
For the new docs site I'd like to avoid keeping a ...
# development
b
For the new docs site I'd like to avoid keeping a static list of backends off somewhere we usually forget to add to. Instead I'd like to "tag" each backend we want documented, and find those when we're ready to generate reference docs. Who wants to bikeshed? šŸ§µ
šŸ’Æ 1
My first (and only, really) idea here is in each relevant 
register.py
, declare 
INCLUDE_IN_PANTSBUILD_DOCS = True
. E.g.
Copy code
# src/python/pants/backend/python/lint/black/register.py
# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

"""Autoformatter for Python.

See <https://www.pantsbuild.org/docs/python-linters-and-formatters> and
<https://black.readthedocs.io/en/stable/>.
"""

from pants.backend.python.lint.black import rules as black_rules
from pants.backend.python.lint.black import skip_field, subsystem

INCLUDE_IN_PANTSBUILD_DOCS = True

def rules():
    return (*black_rules.rules(), *skip_field.rules(), *subsystem.rules())
We'd just use a dead-simple grep to find the relevant 
register.py
files with that exact match (more bikeshedding on the special string) and convert them todotted module names.
c
how about we decide to document all backends that we ship? if you donā€™t want it documented, donā€™t add it to the backends plugin list in the BUILD fileā€¦
b
Either way the choice is separated from the file(s) itself.
c
I kind of like the idea to have fewer places to check when adding a new backend, I think.. šŸ™‚ granted, this is an easy one if itā€™s just a line in register.py, but still..
b
This would be fewer places to check since it's inside the backend you're adding. E.g. if you copy an existing backend it'd "just work" vs remembering to update this file somewhere else
c
.. vs not doing anything (wrgt to docs)
the other point is, that itā€™s ~weird to have a pants internal specific thing here I think.. consider a user looking at our backends for writing an external plugin.. this line could raise the question of ā€œwhat happens if I include this line in my pluginā€, it just feels a little out of place.
so, if we have a reason not to document all backends we ship, then Iā€™m OK with this, but if weā€™ll end up documenting everyone any way, why not just get the list of backends from a pants command for that plugins target?
b
I guess we ought to just source it from the 
BUILD
file that lists backends to ship with. Then we kick the can to that static list
šŸ‘ 1
c
(that list needs some guard rails too btw, as we keep missing to add new backends to it)
yea, trying to keep the number of static things to tweak down..
b
src/python/pants/bin/BUILD
That list looks longer than the docs one
c
well yea, I donā€™t think we currently include all of them in the docs.. the current docs list is from this list: https://github.com/pantsbuild/pants/blob/75888cba9f83a10234abce239ccb199c97d63e3c/build-support/bin/generate_docs.py#L171-L228
b
Right, I wonder if we could (list them all)
c
yea, that was my opening question šŸ˜…
(implying being in favour of doing so)
w
I'm on team "try to document all shipped backends", which ideally means fewer per-backend variables?
āž• 1
b
Actually, i'm starting to lean more towards "pants has an introspection goal" and we use that to know what backends are baked-in. Because right now, the way reference docs work doesn't require running anything in the Pants repo (it only uses off-the-shelf Pants)
But in lieu of that, yeah let's read 
src/python/pants/bin/BUILD
or find all 
register.py
files
c
by off-the-shelf pants, do you mean the off-the-shelf thatā€™s being documented, or used to process the pants repo sources?
if itā€™s the former, you can get the list of backends too from the off-the-shelf pants
buuutā€¦ it doesnā€™t go that far back, if we need to support older versions..?
b
I don't get the choices. But the answer is a Pants that's in a GitHub release. It looks like the easiest would be to recurse the file system, but I think that fails if there's a register file that isn't in the plugin list šŸ™ƒ
c
it shouldnā€™t fail, right? if it does, itā€™s a sign we need to fix it
b
Yes, but it's too late. The release has been made.
Epiphany! Download the (smallest) wheel in the release and use that for the filesystem-source-of-truth! šŸ˜„
šŸŽ‰
Copy code
- name: Get the backends list
        # NB: We need to know what backends are in this release. The easiest way to do that is to find
        # all the `register.py` files in the release. We can't just grep the repo, as some backends
        # might not be included in the release. So the easiest thing to do is download a wheel
        # (we download the smallest one, which is Mac 11) and scrape the backends from it.
        run: |
          curl -L <https://github.com/pantsbuild/pants/releases/download/release_${{> inputs.version }}/pantsbuild.pants-${{ inputs.version }}-cp39-cp39-macosx_11_0_arm64.whl -o pants.whl
          unzip -l pants.whl \
            | grep register.py \
            | awk '{print $4}' \
            | sed 's#/register.py##' \
            | sed 's#/#.#g' \
            | xargs -I{} echo "\"{}\", " \
            | tr '\n' ' ' \
          >> backends.txt
c
if youā€™re ugly like that, Iā€™ll have one that avoids the extra downloadā€¦
b
šŸ‘€
c
ā€¦ šŸš§
Copy code
āÆ pants backends --help-advanced | sed -e '/^[^[]/d' -e 's/\[.*\(pants[^ ]*\) .*/\1/'
b
Oh interesting šŸ˜®
c
b
No, we aren't generating reference docs for dead branches this way
šŸ’Æ 1
c
hmmmā€¦ I wonder ifā€¦.
I got another option tooā€¦ on a similar vein but more robust, hopefully.. šŸš§
b
Copy code
pants help-advanced backends \
            | tr '*' ' ' \
            | grep pants\.backend | awk '{print $3}' \
            | xargs -I{} echo "\"{}\", " \
            | tr '\n' ' ' \
          >> backends.txt
Already looks better
c
itā€™s in 
pants help-all
in json.. just a 
jq
query away
b
Wait I thought that only worked for activated backends
c
just forgot šŸ¤¦ā€ā™‚ļø
no, itā€™s everything
b
šŸ˜®
WTF I just wasted like 2 hours of my life lol
c
the text output you work with now is built off of the json data doh
b
So why do we list backends when doing reference docs today?
c
wdym?
b
c
oh,ā€¦. because the backends list on the help-all is ā€œnewā€
my workā€¦ and Iā€™ve not gotten around to clean up the docs side of it yetā€¦
b
So, just to sonfirm. With no backends activated, now Pants will document all the baked-in backends?
c
yea..
Copy code
"pants.backend.shell.lint.shfmt": {
    "description": "",
    "enabled": true,
    "name": "pants.backend.shell.lint.shfmt",
    "provider": "pants"
  },
  "pants.backend.tools.preamble": {
    "description": "",
    "enabled": false,
    "name": "pants.backend.tools.preamble",
    "provider": "pants"
  },
b
FUCK YEAH
c
enabled or not, itā€™s in the list
AND, it also include in repo plugins if you run this on your own project
b
Right, thats not surprising
c
hmpf
took some work that šŸ˜†
b
How far back does this go?
c
Iā€™ll checkā€¦
b
šŸ˜
Wait, @curved-television-6568 I don't think it outputs all backends by defaults
Copy code
PANTS_VERSION=2.20.0.dev2 pants help-all > 2.20.x.help-all.json
only has one hit for 
add-trailing
. I suspect it's a two-pass approahc because the backends ARE in the 
name_to_backend_help_info
key. So we need to get that first, then use that list to re-run with them all enabled
c
Huh? Not just an issue with missing register.py or something..?
b
Well the hit for 
add-trailing-comma
looks like:
Copy code
"pants.backend.experimental.python.lint.add_trailing_comma": {
      "description": "Autoformatter to automatically add trailing commas to calls and literals.\n\nSee <https://github.com/asottile/add-trailing-comma> for details.",
      "enabled": false,
      "name": "pants.backend.experimental.python.lint.add_trailing_comma",
      "provider": "pants"
    },
So I think it needs to be enabled for it to show up in the other relevant 
help-all
sections
c
Ah, yea.. if ddisabled it will only be in that backends list, yes
Thought that was the list you were relying on
b
Nah, I'm scraping the other usual places because this is generating reference docs
Not a worry though
c
Oh true, you need them all enabled for ref docs.. makes sense
b
2 Views
Open in Slack
PreviousNext
Powered by