https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all, I'm starting to work on a plugin but havi...
# plugins
a
Hey all, I'm starting to work on a plugin but having difficulty figuring out how all the pieces fit together. I've read the docs on Targets, Rules, etc, but it hasn't clicked yet. I could use a hand if anyone is able to shed some light on it...
My short term goal is to: 1. introduce a new target type 2. when built/run/packaged/etc, output a new file alongside the sources.
Eventually I want to query the dependencies and write them to a file in a particular way for use at runtime – but that can come later.
I've got a Target, it's just the regular fields, no special inputs. I've got a 
GenerateSourcesRequest
, but it's unclear to me what the input/output are. For now I've got the input as 
Dependencies
(for future use querying the graph), and an output of 
GeneratedSources
. I've got a Rule that takes the request and returns a 
GeneratedSources
, just creating a 
Digest
, with some file content. I've registered all of these bits in the plugin. This doesn't seem to be running the rule. The target is in the graph and seems to be working, that's all fine. I assume that there must be nothing depending on the output from the rule and therefore not pulling everything through, but I'm not sure how this bit would work.
Am I missing a step here, are there any other parts that should be on my todo list?
Alternatively, is there a good guide for how all the types are supposed to line up through this sort of thing?
w
Have you looked at the tutorials section? There are some resources on the website which are a good place to start: https://www.pantsbuild.org/docs/adhoc-tool https://www.pantsbuild.org/docs/media#tutorials
a
Hey, thanks! I saw the tutorials in the docs, but not the links to external posts. I'm not sure the one in the docs helps me understand where I'm going wrong here though. I explicitly do not want a new goal because I want this generated file to be available in any packaging/running/testing/publishing/etc.
For the package goal, our output seems to reasonably be a BuiltPackage. How did I find this out? I looked at the rules for 
pex_binary
and 
python_distribution
for ideas.
This sounds like it might be quite useful, if I can figure out what the right type should be for my use-case.
w
Another strat would be to find a similar plugin and see how it works in the mainline repo... Sounds like a code gen plugin specifically?
a
Yeah I guess it is. But I've been following the Codegen tutorial and it's not clear how the target is actually run
Also, codegen expects an input, and I'm not sure if I have an input, or what type that would be. For my test run there is actually no input, for the real use-case the input is intuitively 
Dependencies
, but I'm not sure if that's from the right type hierarchy expected at that point.
w
Can you give more details on what it is you're trying to achieve? You want to run any arbitrary command and have something appear in your source files?
a
Right now I'm trying to put "foo" into "foo.txt".
In such a way that when the app runs it could read that file
w
Using which goal?
a
Build, Test, Run, Package, anything that could result in the application needing those files.
w
Got it, and have you looked at adhoc tool? I'm not sure off the top of my head if that would work, but it feels like something I would try first
I just can't recall if the adhoc tool process needs to be the terminal command, or if it can be a dep
a
I have, I think it can be regular code. I'm not sure that would be packageable as a plugin though?
w
Good question, not sure - haven't tried it. So, I've never needed to use codegen just yet - but I'm guessing it sounds like you want to create a target, which is a dependency to your package/run/etc targets, so that it's run along the way ...
a
Would it make sense to create a new synthetic target representing this new file I'm creating, and then inject a dependency on that target in dependees?
I'm guessing it sounds like you want to create a target, which is a dependency to your package/run/etc targets, so that it's run along the way ...
Yes! I think I've got this bit working
i.e. I've currently got: 
python_sources
-> 
foo_target
-> 
pex_binary
, and the binary runs successfully
w
And what's the hiccup? The act of making the file itself?
a
The issue was that the rule code wasn't running
I've changed the return type to 
BuiltPackage
(not sure if that's actually what I need) and now I'm getting graph errors suggesting that there aren't installed rules for generating source files, which seems incorrect.
w
Is this a repo anywhere? Kinda hard to debug
a
Currently no, I'm trying to figure out a minimal version that would be useful to share
I think 
BuiltPackage
is not the right level for me to "inject" the file, because by that point it has already been packaged, but I want to contribute the file to the package.
I guess then what I really need is to add an additional 
resources
target for a generated file, but I'm not sure how to bridge creating a file in a rule, and the resources target.
g
I think 
resource
is the concept you're looking for, maybe? Grep for 
experimental_wrap_as_resources
... that's a source generator going from file "anything" -> resource
a
@gorgeous-winter-99296 thanks! I was actually trying this in a macro up until today. I realised that I really need to be able to query the build graph which means I need a plugin not a macro. I'm not sure how exactly to use existing rules from a plugin though
g
Just import from 
pants
and use the 
Get
helpers as per usual. As long as your 
Get(Output, Input)
matches a rule (or sequence thereof) it'll work. There's a big project to make rules directly callable, I'm honestly not sure where we are around that.
If you want to see how this looks in practice feel free to have a peek at my repos; https://github.com/tgolsson/pants-backends https://github.com/tgolsson/pants-cargo-porcelain/ All of those plugins are up-to-date and usable in production with latest Pants, for some measurement of production 😉
a
Hmm ok, this sounds good
So my rule is going to be from "something" to a... 
Resource
? I don't see a resource type.
Possibly from 
Targets
, because I eventually want to be able to write some data from a subset of targets into a file.
w
Can Resources be transformed along the way? Just as file digests or something
?
g
The way I understand the problem you want a GenerateSourcesRequest from some dummy marker to 
ResourceSourceField
.
a
Ok that makes sense
What depends on the 
ResourceSourceField
?
g
Most things should in terms of running/packaging, if they can... I'm not on a PC so can't do any exact checks.
It's one of the most generic source types, the other one being "file", but file has much fewer semantic uses that I've seen.
a
So the dependency is implicit? i.e. by the rule existing and producing a resource type, the rule will be run because something above it is trying to get "all" the resources?
g
Yes 🙂 Most things when run will do a collect phase where we prepare the sandbox. That should include all dependencies (whatever that means for the target) as well as all sources (of which resource is one type).
a
Ah ha! That was definitely a missing piece of the puzzle for me!
The other bit that's not making sense to me is how the target and my rule are linked. I've got a "Request" class. Does this need to depend on a typed field from the target in order to be "tied" to that target?
g
Here f.ex. https://github.com/pantsbuild/pants/blob/a0ae9500df6b9a05ed1364d76a89f3c4afa4ba09/src/python/pants/backend/shell/shunit2_test_runner.py#L176-L183 You can see the shunit test runner will get all shell sources, generic files, and resources. Including those from codegen. Not all getters will include codegenned sources, for various reasons.
👀 1
👍 1
a
Right now I'm getting a ton of graph errors that seem very generic, possibly because I've added a rule that is too generic and messes up the graph?
g
In general, create custom fields and use those as inputs. If you have no inputs, use a custom field as a marker and use that (probably has to be a sourcefield derivative here)
Rule graph errors suck, so my best advice is to strip out all internal code from the rules that seem to break it... if that works, you've got a goof inside the rule (cycle, bad invocation, ...). If it doesn't work, your rule declaration (or one of the inputs) is goofed, or not registered correctly. Remember to also register your targets 🙂
a
Thanks!
g
A good start 😄 Your generator rule has to return GeneratedSources fwiw, that's likely why it doesn't quite work.
a
Yeah I figured, in the process of figuring out how to do that
I've just realised that Pants is doing AST analysis on the Python rather than running it, and potentially generating graph errors before that point.
Hmm. So no more graph errors, and I'm more confident of the rule being correct now. I think I've got the wrong input/output types. Tried it with a "passthrough" (if that's even a thing?) of the dependencies, and tried it with a resource field set. I'm not sure how GeneratedSources maps to ResourceFieldSet though. I guess that would be through it's own rule?
g
You shouldn't need a ResourceFieldSet in this case, the rule should be ran whenever you run other goals to collect the sandbox inputs. I think the input shouldn't matter here though it probably should be a SourceField derivative for consistency.
a
Thanks for the help. Do you mean for the input or the output? I've tried both and haven't managed to get the rule to run
g
It should be "anything" -> ResourceSourceField iirc... I won't be at a computer properly until tomorrow so can't test running. A good test is to add a raise inside the rule so it's not an error elsewhere and it's actually running.
a
I've already got logging in the rule and set the level so I'm guessing that should be sufficient to see it trigger?
Thanks for the help on this!
Woohoo it works!
So, the input could not be dependencies subclass. I need to dig into why. The output is indeed 
ResourceSourceField
not 
ResourcesFieldSet
which in hindsight is obvious
@gorgeous-winter-99296 thanks so much for your help on this, especially your custom plugins, and thanks @wide-midnight-78598 for the help earlier and your blog posts.
👍 3
g
I haven't verified, but I'm wondering if there is some step assuming that generating sources takes another source as input? In a majority of cases I'm going to guess that is true (protobuf, thrift, etc). Might be worth searching for issues and potentially filing one... it either needs a diagnostic or a bugfix depending on intended behaviour.
h
Glad this worked out! The plugin API is complicated for a reason, but it is complicated...
2 Views
Open in Slack
PreviousNext
Powered by