Apologies if this is the wrong channel. Will move on request. For more context, see: - Issue 23069 - The tail of this slack thread. - This Slack thread.

Support for dynamic rules is through an interface similar to Pants'

@rule

and `collect_rules()`:

@dynamic_rule

and

collect_dynamic_rules()

. For typical cases (i.e., straightforward rules that happen to also require inter-module mutual recursion or otherwise depend on function-internal imports), the user experience is identical: decorate rules that need to be dynamic with

@dynamic_rule

instead, and any

rules.py

files with dynamic rules must call

collect_dynamic_rules()

as part of its

rules()

function (that in turn are called by the plugin's

register.py

). For the indirect-call case, a little more work is required for the author. In our use-case, this extra work is ~trivial. `@rule`s and `@dynamic_rule`s can call each other, and `@dynamic_rule`s support calls with

**implicitly({<...>})

. This has been immediately useful to us, and we are going to continue with this new functionality to support our use case. It is Python-only, and though it depends on existing Pants' Python source, it doesn't need to live in the Pants source code for us to make effective use of it. As far as I am aware, the implementation is not fundamentally "cheating" or otherwise adversely undermining Pants' assumptions, though that may not be true, and otherwise may still be more limited than

@rule

in other subtle ways. Plus the usual caveats about bugs, deficiencies in completeness, etc. Are the Pants maintainers potentially interested in this capability, or its implementation? In order of increasing effort on my part, I could: (1) Do nothing. (2) Provide a brief prose explanation of the enabling mechanisms here, in this thread. (3) Update Issue 23069 with a demo plugin that includes the full implementation for

@dynamic_rule

+

collect_dynamic_rules()

, and demonstrates the capabilities of this infrastructure, so that maintainers can consider the technique and assess whether they want to pursue some version of first-class support for this functionality themselves. (4) Try to put together a pull request that adds

@dynamic_rule

and

collect_dynamic_rules()

, for consideration. (5) Try to put together a pull request that builds this capability directly into

@rule

and

collect_rules()

, for consideration. I understand that this might be beyond or counter to what Pants wants to offer directly, I understand there may turn out to be fundamental flaws to this approach that disqualify it from first-class support even if it's still useful to us specifically, and I understand that even if any of (2-5) sound interesting to a maintainer to consider that there's no obligation to pursue it further. I'd rather avoid more effort than would be reasonably interesting to maintainers, but I'm also happy to pursue a higher-effort item later if it turns out one of the lower-effort items piqued interest in further work (e.g., I could provide (2) on request, and then follow up with (3)+ on request if the mechanism is intriguing enough to maintainers to see a working implementation). Thoughts?

And here's (2), since there's tentative interest, to give some more background and gauge whether there's still interest in case you see obvious flaws in the general case: Background: • Decorator-resolution time (i.e.

@rule

) happens early, and any potential circular imports have not already been resolved. •

_AwaitablesCollector

does its work while a

@rule

is being resolved. It does not visit

import

AST nodes inside functions, because otherwise any function that nevertheless does do function-internal imports to avoid some (non-rule-related) circular import issue would be victims of timing as those import resolutions could fail in the AST visit logic. I.e., were

_AwaitablesCollector

to try to resolve function-local imports at decorator-resolution time, then they're effectively elevated back up to module-level imports susceptible to the same circular import failures they may have been trying to avoid. • Because

_AwaitablesCollector

does not resolve function-local imports due to the timing of its work, any awaitable symbols imported only locally in the function are invisible to the resolution of each

@rule

, causing runtime failure when such a symbol is invoked because the awaitable was never registered at

@rule

-resolution time. This is, to my naive understanding, fundamentally why

@rule

doesn't support cross-module mutual recursion today. Then: The key observation (theoretically! hopefully! seems-to-work-ally!) that enables

@dynamic_rule

is this: we do not actually need to resolve the awaitables associated with the decorated rule at decorator-resolution time. What

@dynamic_rule

does is set up the work needed to generate

TaskRule

and

AwaitableConstraints

objects for the new rule, but actually executing that work is delayed until

collect_dynamic_rules()

is invoked. Because

collect_dynamic_rules()

is called later, during `register.py`/`rules()`-resolution time, module imports have in-general been resolved (or resolved-enough). So, we make an enhanced

class _AwaitablesCollectorDynamic( _AwaitablesCollector )

that simply adds visits for the import nodes that can "see" these circular-import-avoiding symbol imports. With this, a complete set of awaitables can be determined, even for rules making cross-module recursive calls. Again, I don't know if there are fundamental caveats to this that make it unworkable in the general case (i.e., integration with

@rule

) or unworkable even as a distinct

@dynamic_rule

. I am not an expert on the breadth of magic that the Pants engine provides through its rules. But it does seem to work, and support all the functionality we've needed so far, most notably

**implicitly()

and the fill-in-the-blanks argument magic in general.