https://pantsbuild.org/ logo
#general
Title
# general
h

hundreds-father-404

05/13/2022, 11:42 PM
I started adding Sphinx support 🙂 Thoughts on this target modeling?
1
💯 1
🙌 5
Background • We need to support multiple distinct docs "projects" in Pants, i.e. site1 vs site2 • It seems like a docsite is pretty atomic. You must have the
conf.py
file for settings, and files within a site can easily depend on each other Proposal: • Use a target type like
sphinx_project()
, whose sources include the
conf.py
and all
.rst/.md
files in the directory and recursive subdirectories • To include Python source code, you'd add to the
dependencies
field (or we infer) Alternative: • File-level modeling,
sphinx_source
vs
sphinx_sources
Benefits of
sphinx_project()
• Less boilerplate • Easier for us to implement, e.g. no dep inference needed • Obvious how to support docs from Python source code Downsides of `sphinx_project()`: • not good project introspection. You can't use Pants to query which pages depend on which others. • couldn't say to build a single file rather than whole project (at least w/ current technology
Pretty sure
sphinx_project()
is the way to go, but curious for feedback before spending more time writing this
h

high-yak-85899

05/13/2022, 11:46 PM
I like less boiler plate
1
👍 1
h

hundreds-father-404

05/13/2022, 11:48 PM
me too. And it seems less likely you're gonna be setting metadata on an individual file. Unlike Python, you're not really saying "this particular file has a test timeout of 20s" Sphinx seems pretty fast from playing with it, so not a huge deal if dependencies aren't precise. That'd be good to confirm from Sphinx users tho
b

bitter-ability-32190

05/13/2022, 11:54 PM
Sounds like a start! Any support is better than none. And it can always be improved later. And by using the feedback from initial implementation
h

high-yak-85899

05/13/2022, 11:57 PM
I added some stuff to our sphinx building to monkeypatch the generated rst code to change up the depth of table of contents it shows and such. Sphinx really doesn't like namespace packages (we had to add dummy
__init__.py
modules throughout if a directory didn't contain any python modules). Other than that, we never needed to do much. It was all handled through
conf.py
and associated extensions.
👍 1
h

hundreds-father-404

05/13/2022, 11:59 PM
Okay yeah I like this
sphinx_project
target type a lot! For example, we can add a field for what mode(s?) you want to generate in, e.g. HTML. So you change on a per-project basis
👍 1
r

rhythmic-morning-87313

05/14/2022, 4:07 AM
i think we should take care of autodocs and python-source dependencies from the docs, but practically this type of caching/incremental build also could be offloaded to Sphinx itself.
👍 2
c

curved-television-6568

05/14/2022, 5:55 AM
Can't the sphinx project generate file level targets. That keeps the boilerplate down while still being open to file level metadata via overrides.
h

hundreds-father-404

05/14/2022, 2:00 PM
Can't the sphinx project generate file level targets.
Possibly. I think I'm gonna start w/ just
sphinx_project
for now, then we could probably add if there's demand
👍 1
n

nice-florist-55958

06/15/2022, 8:53 PM
We built some scripts to work alongside Pants in the build chain for when it came to the documentation build step, so this would be a great feature to have to ultimately get rid of those. Is the
package
goal on a
sphinx_project
expected to be similar to invoking the Makefile that comes with every Sphinx project? Would the output type be specified in the goal's args or the target's params (e.g., "make html", "make pdf")? Not sure if this would affect the design choice, but we also make heavy use of the IPython Sphinx extensions module that allows your
.rst
to drop into an IPython kernel, do some Python, and then save the outputs/plots in a way that a user would experience if they had a console or notebook open. In our build chain we had to make sure that any given project on the repo had a correctly setup virtual env for their docs sub-project to be built correctly and make this type of documentation possible. That turned out to be a bit of a headache, esp. for first-party code (maybe because we have become so accustomed to Pants doing this so elegantly in other situations!); hopeful whatever becomes of this new target, these sorts of things are easily handled :)
👍 1
h

hundreds-father-404

06/15/2022, 9:03 PM
Hi! The output type would be in the target's fields. And yes,
package
is like invoking
make
to build it I ended up reverting Sphinx in 2.13 because CI was flaky and the design needs more work. But, it's a foundation and I hope to pick it back up (I want to use Sphinx for Pants itself)
👍 1
n

nice-florist-55958

06/15/2022, 9:14 PM
I hope it's successful when you do! When the time comes and you need some early adopters for testing, let me know.
❤️ 1
h

hundreds-father-404

06/15/2022, 9:18 PM
Will do! And lmk if you have any interest in implementing, we're definitely always happy to help contributors
❤️ 1
n

nice-florist-55958

06/17/2022, 9:29 PM
We are planning at some point to commit time to exploring Pants’ internals with the aim of being able to contribute changes we need in the future, but I am not sure how far we will get (to an extent it depends on how institutionalized Pants becomes within our org—I'm trying to make a good case for it over Bazel and our own horribly inadequate dynamic language build tools). You or someone else will likely beat us to the documentation goal, but we'll see ;) One of the first things we’d want to do is add support for the Q programming language (very common for projects to use both or spin up a Q service that the Python interacts with) since it is so widely used here (and at banks/hedge funds in general).
👍 1
2 Views