https://pantsbuild.org/ logo
#development
Title
# development
b

bored-art-40741

06/20/2021, 6:28 PM
I'm trying to understand a Pants target/rule API behavior here; I'm not sure if this is deliberate and I'm misunderstanding the intent, or if it's not how Pants is supposed to behave. So I have a very basic
Target
subclass, JavaLibrary. When I use the
java_library
alias in a BUILD file with two Java sources present (using the default pattern to capture all non-test
.java
sources), I expect a single
JavaLibrary
target to be produced in the build graph. But instead, there are 3 targets:
//:lib
, which depends on
//ExampleLib.java:lib
and
//OtherLib.java:lib
(corresponding to the two source files). I can sort of see based on my past experience with Pants why we might be generating "synthetic" targets in this situation to more cleanly encapsulate source files, but here's the catch: the two "synthetic" targets depend on each other (circularly). This doesn't appear to directly upset the build graph, but it does cause my rule to break since it introduces a cycle in the rule graph. Am I using this API wrong, or is this just something that hasn't been an issue for Pants yet because the Python rules don't have to distinguish between direct and transitive dependencies much?
So I guess everything here makes logical sense, in more-or-less the way I guessed and for the reasons I guessed, but I still don't understand how this doesn't cause cycles everywhere, or why this doesn't anger the graph's cycle detection earlier
OK, I've found a "solution", although I'm not very sure it's a good one: instead of requesting
Targets
in my
DependenciesRequest
, I ask for
ExplicitlyProvidedDependencies
. This bypasses
Subtarget
generation (among other things), and ends up behaving how I expect.
w

witty-crayon-22786

06/21/2021, 4:08 PM
right: so this behavior is “file level dependencies by default”. we anticipated that there might be some extra work involved for compiled languages, but didn’t fully design it.
what we expected was that at compilation time, we’d compile entire cycles as a single compilation unit
so you get a M:N relationship between targets and compiler invokes
there is some backstory in https://github.com/pantsbuild/pants/issues/10455 … but in short: we always generate file-level targets (“sub” targets), and we attempt to operate on them by default
b

bored-art-40741

06/25/2021, 11:53 PM
So I've been mulling a bit more since our meeting earlier this week Stu, and it does seem like for improving the javac rules we're probably going to need to have the "coursening" step available. I can request
Targets
instead of
ExplicitlyProvidedDependencies
, and then compile at the per-file level, but I'm immediately going to run into issues with subtarget cycles at that point. Do you have a feel for the level of difficulty for adding support to request the coursened graph as a product?
w

witty-crayon-22786

06/26/2021, 12:25 AM
Shouldn't be too bad...? The only thing that might get in the way is figuring out the right API
I'll take a look this weekend.
b

bored-art-40741

06/26/2021, 12:33 AM
I guess what I'm asking is if this is something that I should start looking into, or if it's something you expect to take a shot at in the next few weeks. If the former, I'm happy to get started, but it'll obviously take me longer to ramp on that part of the engine. If the latter, I can just build around the assumption that it'll arrive relatively soon
w

witty-crayon-22786

06/26/2021, 1:17 AM
I think I can commit to getting it done this weekend. Not sure when though.
Wasn't able to promise anything yesterday because of the talk, and currently pretty exhausted. But I'm excited to take a look at it.
b

bored-art-40741

06/26/2021, 1:44 AM
Understood, not trying to press for it that quick, just trying to get a feel for timeline!
Hmm, so I'm trying to set up some basic tests to request
Targets
and work from there, and I discovered
UnparsedAddressInputs
, which has a docstring that suggests it does exactly what I want in a test:
Copy code
await Get(Targets, UnparsedAddressInputs([...], owning_address=Address("original"))
Unfortunately, I can't convince
RuleRunner
to do this, and I can't find a missing rule
Copy code
targets = rule_runner.request(Targets, [UnparsedAddressInputs(["//:lib"], owning_address=None)])
Copy code
E       Exception: No installed QueryRules can compute Targets given input Params(UnparsedAddressInputs), but it can be produced using:
E         Params(Addresses)
I've tried going down the rabbit hole of getting
Addresses
first, but it seems to be leading me in the wrong direction:
Copy code
E       Exception: No installed QueryRules can compute Addresses given input Params(UnparsedAddressInputs), but it can be produced using:
E         Params((OptionsBootstrapper, Specs))
RuleRunner
seems to already have an
OptionsBootstrapper
, so this is where I'm stuck
I'm especially confused because graph.py seems to have a very straightforward set of rules that transform
Addresses
into
UnexpandedTargets
into
Targets
Ahhh there it is, the issue is with
UnparsedAddressInputs
->
Addresses
, not the rest
Doh, the solution was much simpler than I was giving it credit for, after a lot of fruitless digging: add
QueryRule(Targets, [UnparsedAddressInputs])
👍 1
For some reason I thought that as long as the rules were provided to the
RuleRunner
, it would transitively hook up
A -> C
if it had
A -> B
and
B -> C
, but it doesn't
w

witty-crayon-22786

06/27/2021, 1:13 AM
I made some progress on the coarsening API today but it's not quite ready to post we'll try to post something before end of day tomorrow
👍 2
b

bored-art-40741

06/27/2021, 1:44 AM
I also made progress on simple Java dep inference, I think I should be pretty close to an MVP tomorrow
🤘 2
w

witty-crayon-22786

06/27/2021, 5:57 PM
posted a really rough draft of the target coarsening: https://github.com/pantsbuild/pants/pull/12251 … the test should show what it does.
it’s definitely not efficient, but if you’re on a tighter timeline to land dependent stuff, it’s probably sufficient for now.
b

bored-art-40741

06/27/2021, 7:17 PM
Sweet, I'll take a look here shortly
w

witty-crayon-22786

06/28/2021, 9:57 PM
oh, also: rewinding waay back to your original question: all file targets within a target depend on one another if there are no inference providers for the sources… so i think that your false cycle will actually go away when you’ve implemented inference (depending on whether you have all sources implicitly depend on files in their package…?). nonetheless, some real cycles can of course be produced, and that’s where coarsening comes in.