#18235 A generic "generate" goal for version-controlled generated files New discussion created by cognifloyd For a variety of reasons (many of which could probably be summarized as "legacy"), some generated files need to be committed to version control. Since these files need to be in version control, it would be great if pants could make it easier to manage them. Here are the goals I have for whatever solution we use: • Generated files should be up-to-date. • If the files are out-of-date (eg one of the inputs used to generate it has changed), then the user needs an error or warning with an explanation of how to re-generate the files. A survey of current pants goals goal:

export-codegen

At first blush, codegen seems ideal for making generated files. But codegen in pants is not designed for version controlled files. When pants generates code, it is considered an internal by-product as described in the PR that introduced the

export-codegen

goal:

It's important that we write to

dist/

, rather than saving to the build root, as these are generated files that should be kept out of version control. Also, we'd have major issues if Pants starting thinking those generated files were actual production files, e.g. a [target]'s default sources field claiming ownership of those files.

The

export-codegen

goal makes these generated files available in

dist/

for those who want to:

use the results of codegen outside of Pants, including:

1. Inspect the files for debugging purposes.

2. Save the files for IDEs to use.

So, we need a different form of codegen in pants. goals: lint, fmt, fix Version controlled files are the bread and butter of

lint

,

fmt

, and

fix

which is opposite of

export-codegen

which is for files that are not version controlled. Fixers and formatters also run during the

lint

goal so that lint fails if they need to fix or format a file. Code generation of files that are version controlled is remarkably similar to

fmt

and

fix

. Files that need to be changed are advertised via the

lint

goal. Logically, fmt and fix are very different than codegen - but the process, the UX, and the underlying rules seem quite similar to me. In the StackStorm project, I've shoved my (version controlled) file generation under the

fmt

umbrella even though it's not really a formatter. Logically this generation is also not a fixer. But, one of the benefits of doing this is that: lint will fail if the generator needs to regenerate the file(s) and it will provide an error message that explains how to run

./pants fmt ...

to resolve the lint error. I've heard of others that create a "test" that fails if the generated file(s) were not properly regenerated. I really like using

lint

for this to take advantage of an the fine grained caching and reusing the cache for both lint and fmt/fix. goal: generate-lockfiles Another goal that is very similar to the file generation feature I'm looking for is the

generate-lockfiles

goal. The warnings/errors saying to regenerate can be triggered in many places where rules need to use the lockfiles. So, there is no need to integrate that warning in a lint backend/rule. The goal also provides a fairly ergonomic interface for regenerating those. generate-lockfiles looks up the resolves, translates them into lockfile paths, and then generates the lockfile at that path. Proposal for a new "generate" goal If there were a generic "generate" goal, it could go in the opposite direction of `generate-lockfiles`: given the path of a lockfile, look up the resolve associated with that lockfile, and then generate the lockfile at that path. Similarly, that "generate" goal could take a path to another generated file, look up owning target for it, and if that target is explicitly for generated files, run the plugin/subsystem/tool that encompasses the codegen logic for that target. This goal could be only for non-lockfile generated files. If we did include lockfile generation in this goal, then the owning target would probably be a synthetic target synthesized from the config in pants.toml -- so it is explicitly defined (even though not defined in a BUILD file). This goal should be modeled on fix/fmt (not modeled on export-codegen). This would extend the lint request/result classes (like fix/fmt do) so that file generation happens in a sandbox during lint, failing lint if something should be regenerated. When lint fails here, the error message would prompt the user to run

./pants generate path/to/generated/file/or/dir

. For a plugin to provide this generate rule, it would either have to add a custom target, or add a field to an existing target to allow flagging it as owning generated files. This would be a codified assumption built into the

generate

goal: all generated files must be owned by exactly 1 file generating target: If unowned, they will not be materialized in the workspace; If owned by more than one file generating targets, raise an error. Generating a file that matches a file glob for that target counts for this. Then, if any other linters can work with a generated files the standard

skip_tool

fields provide an escape hatch if the generated code does not need to meet the same standards as other files. Or, for StackStorm, I can add an extra linter specific to the generated file that checks more than just whether or not it needs to be regenerated. Goal: run A lot of work has gone into runnable targets like

adhoc_tool

and

shell_command

. If combined with the generate goal, those could allow repos to use light-weight run targets and file generator targets instead of writing a medium- or heavy-weight pants plugins. So, a runnable target could be tied to the file generating target such that the adhoc_tool or pex_binary is used to actually do the generation when a user does

./pants generate path/to/generated/file

. For the

run

goal, the UX focuses on the generator script/target. For this "generate" goal, the UX focuses on the files to generate. So they are complementary. goal: go-generate #16909 This seems very similar and could perhaps be rolled into the new "generate" goal. go-generate references the go modules that define the generation. I'm looking at something that targets the generated files. But, often go:generate commands work within the same module where they are defined, so maybe it is effectively the same thing. But, this is something that would not run via Lint. So, that would probably be a go backend option to skip running for lint is the goals were combined. discussion Actually naming the new "generate" goal is a different discussion. When we discuss that, we'll need to make sure it can be distinguished from the codegen that already exists in pants. So, would a generic generate goal be helpful for anyone else? Does a generate goal that follows a process similar to fix/fmt make sense for other projects? pantsbuild/pants