I'm trying to build a PoC that generates documentation (with mkdocs) for a handful of python projects in a monorepo before packing the generated html files into an nginx docker image. The html is entirely auto-generated, but for the docker build to work inside of the pants sandbox, those generated files need to be dependencies of the image. I set up an

adhoc_tool

target to do this, since the documentation generator is a python package, but then I realized that it would need in its

execution_dependencies

all of the python source files, some docs folders which currently aren't targets (but could be made targets), some yaml files for configuration in each repo... Am I approaching this in the right way, or is there some more straightforward way to go? Could obviously do it out-of-band without touching pants, but then I also couldn't very easily take the generated html files and copy them into the docker image without somehow telling pants about them. Any patterns for documentation generation more generally?

First, I misunderstood that I could put source files in more than one target and that it's easy to create targets based on globs like "all the python files", or "all the markdown files". So that part is easy and fine.

👍

So even though I have the extension packages in my requirements and they're included in

execution_dependencies

for the

adhoc_tool

, mkdocs still doesn't seem to know that they exist. Everything works fine in the exported virtual environment.

Can you share the

adhoc_tool

target definition you're using?

Sure, it looks like this.

adhoc_tool(
    name="mkdocs_build",
    runnable="3rdparty/python:reqs#mkdocs",
    args=["build"],
    execution_dependencies=[
        "3rdparty/python:reqs#mkdocs-material",
        "3rdparty/python:reqs#mkdocs-monorepo-plugin",
        "3rdparty/python:reqs#mkdocstrings",
        "//:all_python_sources",
        "//:all_docs"
    ],
    output_directories=["dist/site",],
)

The

mkdocs build

command runs, and the errors that I'm seeing are generated from mkdocs, not pants. Things like:

ProcessExecutionFailure: Process 'the `adhoc_tool` at //:mkdocs_build' failed with exit code 1.
stdout:

Aborted with a configuration error!

stderr:
ERROR   -  Config value 'theme': Unrecognised theme name: 'material'. The available installed themes are: mkdocs, readthedocs

But this error isn't present when running in an exported virtual environment, and in that virtual environment the mkdocs cli command does list "material" as one of the available themes. That's why it seems like it might have to do with how pants detects/resolves entry-points for plugins, but I'm not sure.

Ah, I think execution_dependencies may be making a executable called

reqs#mkdocs-material

available on the

PATH

, as an independent "binary" (that just happens to be implemented in Python under the hood), rather than integrating it with the Python venv that the main mkdocs executable is running from. You may find more success by first building mkdocs and the other packages into a

pex_binary

, and then use that as the runnable. This'll mean they end up in the one venv. Maybe something like:

pex_binary(
    name="_mkdocs_pex",
    dependencies=["3rdparty/python:reqs#mkdocs", "3rdparty/python:reqs#mkdocs-material", "3rdparty/python:reqs#mkdocs-monorepo-plugin", "3rdparty/python:reqs#mkdocstrings"],
    script="mkdocs", # whatever the script entry point in mkdocs is called
    # This pex is just used to group up all the required libraries into a single venv
    # for immediate execution below, so we don't need it as a single file and choose
    # the cheapest layout:
    layout="packed", 
)

adhoc_tool(
    name="mkdocs_build",
    runnable=":_mkdocs_pex",
    args=["build"],
    execution_dependencies=[
        "//:all_python_sources",
        "//:all_docs"
    ],
    output_directories=["dist/site",],
)

Thanks so much! This ended up being exactly what I needed. And that helps my mental model quite a bit. I was thinking for some reason that the adhoc_tool would be more of a virtual environment when that's kind of the whole point of pex... I really appreciate it!

@stale-rose-15535 could you share more examples about your mkdocs setup?