Hi. Just getting started with Pants and need to write some custom rules. I'm a little confused about how to chain rules together. For example, suppose my rule consume target

my_target

and generates some new files. As I understand it, I can return a custom`Goal` subclass that contains some information about those files but that does not generate new targets. Now suppose I want to make sure my generated code is safe via

bandit

. Since I don't have a new target and

bandit

's

@rule

doesn't take a

MyCustomGoal

how do I tell

bandit

that it depends on (or should check) these custom files?

Hi Oliver, welcome! Happy to help. Taking a step back, it sounds like your plugin is trying to generate Python from some other format, and then you want Bandit to run on that generated code. Is that right?

Yea, that was the quesiton. It's actually a bit more complicated than that but I figured the easy way to ask the question would be to pretend that was the case. If I could do python gen -> bandit run I think I could solve my actual issue.

Are you specifically trying to run Bandit, which is something already written? Or are you in total control of the whole pipeline and trying to run a tool that you control?

I am in control of the total pipeline. That was just an example. But conceptually I understand that if a rule,

A

, that returns a

Foo

you can have another rule,

B

, that requests a

Foo

as input so when that goal

B

is requested Pants knows it need to run the rule

A

first. What I don't understand is the more general case like

bandit

where it can't know what generated files might exist beforehand. If I generate a file how can I "talk about it" in other targets and rules?

rules don't output targets but they do generate files

To clarify, rules can output whatever they want (as long as it's a single type with a hash and eq implementation). Rules can output targets, `Digest`s representing files, some custom type, etc

specify that output should be the input to another target or rule

You either put that output type as an argument to your

@rule

function, or use

await Get(OutputType, OutputTypeRequest)

in the calling rule. Which format you use depends on if Pants needs some input in order to compute

OutputType

vs. if it can be computed without extra input from the callsite

Hmmm... I think I understand what you're saying but I'm still missing something. For example, suppose I have a

blob.foo

file in

some/dir

and my custom rule generates a

blob.py

from that. If I want something else to depend on

blob.py

how do I refer to it? Can I put a

some/dir/blob.py

in the

dependencies

section of something like a

package

or

bandit

rule? I suspect not 'cause I don't think Pants wold know that file exits unless I specifically

pants my_rule some/dir/blob.foo

, right?

What I don't understand is the more general case like bandit where it can't know what generated files might exist beforehand.

The caller needs to know about the output type so that it can request the output type There is a way to make that more generic through a thing called unions, which are polymorphism for the engine. This is how all of Pants's plugin hooks work - we have no idea what plugins users might right, but unions allow us to handle that. Although, most the time plugins don't need to set up their own unions because you are usually in total control of your pipeline https://www.pantsbuild.org/docs/rules-api-unions

I think the union rules is maybe the only section of the API docs I haven't read 🙂 I can give that a read.

With codegen, is that only an example or you do want to use codegen in your plugin? If so, there is already prior art for how to do that, including a plugin hook https://www.pantsbuild.org/docs/rules-api-and-target-api#enabling-codegen https://www.pantsbuild.org/docs/plugins-codegen

Rules can output targets

They can? I thought the pattern was:

class MyGoalSystem(GoalSubsystem):
    # ...

class MyGoal(Goal):
    # ...

@goal_rule
fun my_rule(...) -> MyGoal:
   # ....

that is, I thought

MyGoal

had to be a subclass of

Goal

and my

@goal_rule

had to return that.

Ah, so that is only if you want to register a new goal, like a new verb that people can run. A lot of plugins never add new goals and ~90% of rules are not goal rules With goal rules, though, indeed they must return a subclass of

Goal

got it!

You're welcome! There is certainly a learning curve, and also that is only the first version of those docs - we know we want to improve them, so your feedback is helpful

I'm coming from Bazel so have some familiarity with the concepts. Finding your docs to be better than theirs.