https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to use globs in dependency declarat...
# general
f
Is there a way to use globs in dependency declarations? Like... perhaps via macro? Context in 🧵
I'm trying to group files by "component" in the context of functional acceptance testing. The tests won't actually have import dependencies on these files, since they'll be packaged and launched in VMs elsewhere, and the tests only call them via API.
h
No. The closest there is using a generic 
target
as a "bag of dependencies": https://www.pantsbuild.org/v2.10/docs/reference-target Honestly, address syntax has gotten so complex and is going to get even more complex with the upcoming parametrize feature that I'd be really hesitant to add even more variants on address syntax. We believe the complexity is a necessary evil, but still https://www.pantsbuild.org/v2.10/docs/targets#target-generation
f
I get that
h
Not that it will necessarily solve this particular issue, but the other angle is to continue lowering the times you need to explicitly use addresses. Two recent examples: • @bitter-ability-32190's great work on asset inference https://github.com/pantsbuild/pants/pull/14049 • codegen is now less verbose: https://github.com/pantsbuild/pants/pull/14691
f
I mean, I suppose I could write a plugin to do this
As long as there's a rule for getting files from globs...
ah... or I could just use tags and 
overrides
h
As long as there's a rule for getting files from globs...
Yeah, although here you probably want the addresses of those file globs right? The plugin hook for dependency inference expects you to return 
Addresses
You can use 
Get(Owners,  OwnersRequest)
ah... or I could just use tags and overrides
How so?
I mean, I suppose I could write a plugin to do this
How frequent will this problem be? Are you adding a new testing framework it sounds like, that might be replicated in multiple places? If so, a plugin could be a really good idea. You spend some time to make things much more intuitive and automatic for your org's users
f
Frequent enough to justify it as a plugin if we do it more, but that's still an if. I'm just looking for a fast option for a PoC right now
👍 1
I think I can cheat and make tags that are named like a component address
Copy code
python_sources(
    name="aiven",
    sources=["aiven/**/*.py"],
    overrides={
        "aiven/admin/*_cli.py": {
            "tags": ["components/admin-commands"],
        }
    }
)
and then 
components/admin-commands/BUILD
can be a bare 
target()
that I can make the tests depend on
h
how do you go from that 
tags
into it pulling in dependencies? 
tags
isn't wired up to 
dependencies
unless you add a plugin
f
filter
+ 
peek
lol this is getting messy
h
oh yeah because this is only piping things into your custom test runner. That could work, but do note that 
./pants dependencies
will be lying and 
--changed-since
too For quick and dirty, I'd recommend using 
target()
as a bag of dependencies For robust and hopefully not too slow, I'd recommend the plugin. Lmk if you're interested in this route, I think it'd only be about 20 lines of code
f
./pants --changed-since=main list | xargs ./pants filter --tag-regex="^components" | ./pants peek | jq uniqe(.tag)
👀 1
ugh yeah idk
h
Lmk if you're interested in this route
I can try to find some examples you can crib if you want. when redesigning 
example-plugin
repo, we should definitely have examples of this plugin hook Might also be helpful to show off a pants plugin to coworkers
👍🏻 1
f
I think as long as I can get this to work no one cares haha
well my teammates would care actually
seems like a plugin would just need a custom field 
glob_dependencies
or something like that, and then I could define targets that use that like a barebones 
component
target
h
yeah that's totally feasible! You would: 1. Use a plugin field to add it to what you want: https://www.pantsbuild.org/v2.10/docs/target-api-extending-targets 2. You can use "dependency injection" plugin hook rather than "dependency inference". tl;dr is that dependency injection does not require a 
sources
field. See https://github.com/pantsbuild/pants/blob/fe41887641970a34dc7de19d5c49442978ec83e1/src/python/pants/backend/codegen/thrift/apache/python/rules.py#L66-L94 for an example rule. Use 
inject_for = Dependencies
rather than something more specific, which means this will get used for every single target 3. In your rule, also use the 
WrappedTarget
line. Then 
tgt.get(DependenciesGlobField)
. Now use that to call 
await Get(Owners, OwnersRequest())
4. Register the 
UnionRule
✔️ 1
Btw, I hadn't thought of the idea of a new field. That's a great idea! That makes me much more open to Pants adding this feature because you avoid making the syntax of 
dependencies
even more complex Still an open question if it's desirable to add, but cool!
f
cool, i'll give it a try
❤️ 1
h
FYI this will not play nicely with the upcoming 
parametrize
feature meant to make multiple resolves (lockfiles) ergonomic. For example:
Copy code
python_source(
  name="utils"
  source="utils.py",
  resolve=parametrize("python-default", "data-science"),
)
That is syntactic sugar for creating two targets: 
dir:utils&resolve=python-default
and 
dir:utils&resolve=data-science
. It's important that you don't depend on both because Pants won't like that you're attempting to use two resolves in the same closure. A glob of 
dir/utils.py
would match both targets as they're both the owners. Now, you could make your plugin hook fancy and proactively filter out targets from 
Owners
that it doesn't like. Totally feasible. Only holdup is that it means we would have a harder time generalizing this to pantsbuild/pants as a field on every single target cc @witty-crayon-22786
f
I don't think we'll be using multiple resolves for quite a while
👍 1
Okie doke, took me a while to get around to this, but it works pretty decently: https://gist.github.com/jriddy/b3f8341033d8b16d18b4feb4945d322d
❗ 1
👀 1
Open in Slack
PreviousNext
Powered by