https://pantsbuild.org/ logo
Join Slack
Powered by
I'm trying to understand a Pants target/rule API b...
# development
b
I'm trying to understand a Pants target/rule API behavior here; I'm not sure if this is deliberate and I'm misunderstanding the intent, or if it's not how Pants is supposed to behave. So I have a very basic 
Target
subclass, JavaLibrary. When I use the 
java_library
alias in a BUILD file with two Java sources present (using the default pattern to capture all non-test 
.java
sources), I expect a single 
JavaLibrary
target to be produced in the build graph. But instead, there are 3 targets: 
//:lib
, which depends on 
//ExampleLib.java:lib
and 
//OtherLib.java:lib
(corresponding to the two source files). I can sort of see based on my past experience with Pants why we might be generating "synthetic" targets in this situation to more cleanly encapsulate source files, but here's the catch: the two "synthetic" targets depend on each other (circularly). This doesn't appear to directly upset the build graph, but it does cause my rule to break since it introduces a cycle in the rule graph. Am I using this API wrong, or is this just something that hasn't been an issue for Pants yet because the Python rules don't have to distinguish between direct and transitive dependencies much?
So I guess everything here makes logical sense, in more-or-less the way I guessed and for the reasons I guessed, but I still don't understand how this doesn't cause cycles everywhere, or why this doesn't anger the graph's cycle detection earlier
OK, I've found a "solution", although I'm not very sure it's a good one: instead of requesting 
Targets
in my 
DependenciesRequest
, I ask for 
ExplicitlyProvidedDependencies
. This bypasses 
Subtarget
generation (among other things), and ends up behaving how I expect.
w
right: so this behavior is “file level dependencies by default”. we anticipated that there might be some extra work involved for compiled languages, but didn’t fully design it.
what we expected was that at compilation time, we’d compile entire cycles as a single compilation unit
so you get a M:N relationship between targets and compiler invokes
there is some backstory in https://github.com/pantsbuild/pants/issues/10455 … but in short: we always generate file-level targets (“sub” targets), and we attempt to operate on them by default
b
So I've been mulling a bit more since our meeting earlier this week Stu, and it does seem like for improving the javac rules we're probably going to need to have the "coursening" step available. I can request 
Targets
instead of 
ExplicitlyProvidedDependencies
, and then compile at the per-file level, but I'm immediately going to run into issues with subtarget cycles at that point. Do you have a feel for the level of difficulty for adding support to request the coursened graph as a product?
w
Shouldn't be too bad...? The only thing that might get in the way is figuring out the right API
I'll take a look this weekend.
b
I guess what I'm asking is if this is something that I should start looking into, or if it's something you expect to take a shot at in the next few weeks. If the former, I'm happy to get started, but it'll obviously take me longer to ramp on that part of the engine. If the latter, I can just build around the assumption that it'll arrive relatively soon
w
I think I can commit to getting it done this weekend. Not sure when though.
Wasn't able to promise anything yesterday because of the talk, and currently pretty exhausted. But I'm excited to take a look at it.
b
Understood, not trying to press for it that quick, just trying to get a feel for timeline!
Hmm, so I'm trying to set up some basic tests to request 
Targets
and work from there, and I discovered 
UnparsedAddressInputs
, which has a docstring that suggests it does exactly what I want in a test:
Copy code
await Get(Targets, UnparsedAddressInputs([...], owning_address=Address("original"))
Unfortunately, I can't convince 
RuleRunner
to do this, and I can't find a missing rule
Copy code
targets = rule_runner.request(Targets, [UnparsedAddressInputs(["//:lib"], owning_address=None)])
Copy code
E       Exception: No installed QueryRules can compute Targets given input Params(UnparsedAddressInputs), but it can be produced using:
E         Params(Addresses)
I've tried going down the rabbit hole of getting 
Addresses
first, but it seems to be leading me in the wrong direction:
Copy code
E       Exception: No installed QueryRules can compute Addresses given input Params(UnparsedAddressInputs), but it can be produced using:
E         Params((OptionsBootstrapper, Specs))
RuleRunner
seems to already have an 
OptionsBootstrapper
, so this is where I'm stuck
I'm especially confused because graph.py seems to have a very straightforward set of rules that transform 
Addresses
into 
UnexpandedTargets
into 
Targets
Ahhh there it is, the issue is with 
UnparsedAddressInputs
-> 
Addresses
, not the rest
Doh, the solution was much simpler than I was giving it credit for, after a lot of fruitless digging: add 
QueryRule(Targets, [UnparsedAddressInputs])
👍 1
For some reason I thought that as long as the rules were provided to the 
RuleRunner
, it would transitively hook up 
A -> C
if it had 
A -> B
and 
B -> C
, but it doesn't
w
I made some progress on the coarsening API today but it's not quite ready to post we'll try to post something before end of day tomorrow
👍 2
b
I also made progress on simple Java dep inference, I think I should be pretty close to an MVP tomorrow
🤘 2
w
posted a really rough draft of the target coarsening: https://github.com/pantsbuild/pants/pull/12251 … the test should show what it does.
it’s definitely not efficient, but if you’re on a tighter timeline to land dependent stuff, it’s probably sufficient for now.
b
Sweet, I'll take a look here shortly
w
oh, also: rewinding waay back to your original question: all file targets within a target depend on one another if there are no inference providers for the sources… so i think that your false cycle will actually go away when you’ve implemented inference (depending on whether you have all sources implicitly depend on files in their package…?). nonetheless, some real cycles can of course be produced, and that’s where coarsening comes in.
Open in Slack
PreviousNext
Powered by