https://pantsbuild.org/ logo
Join Slack
Powered by
Wait, if we have it so that a file dependency auto...
# development
h
Wait, if we have it so that a file dependency automatically depends on every single other file in its original owning target, doesn’t that mean that we’ve defeated the purpose of file dependencies? The result of 
dependencies --transitive
will be that that single file depends on every single thing the original owning target depended upon.
w
you would only do it if no inference
have been thinking about this for a few minutes, so one sec.
h
But the point of the bug I found in Toolchain is that even with inference, things are breaking. That test file depends on its siblings via Pytest magic, and we can’t infer that because there are zero import statements
(Pytest magic being a Pytest parameter coming from another file, I think 
conftest.py
)
w
Copy code
if inference:
	subtargets do not depend on one another automatically
	can be made to depend on one another via a self-dep
else:
	subtargets depend on one another
(self dep being literally the base target having a declared dep on itself)
(or, as an alternative to the self-dep, the whole target could depend on one of the files in the target, if need be… that’s fine too afaict.)
h
a declared dep on yourself seems... odd. That will be hard to explain to users.
👍 1
w
@happy-kitchen-89482: i’m not sure about that. the fundamental change at the core of this is: “targets are metadata applied to files”
that part gets lost periodically in the feature that we are currently implementing, but it is completely consistent in light of that.
a self dep in a target means “all of these files depend on all of these files”. a file dep on a target means “all of these files depend on this file”
h
How do you handle “I have 5 files. A depends on B. Nothing else depends on B.” Currently, in your BUILD file, you would need to put “B” in the 
dependencies
for everything, even though 4 files don’t want it
w
you could create more targets.
i expect that that will not be worth the boilerplate for most people though.
hey folks: Eric and I each wrote up some thinking about this: interested in thoughts (although not urgently, as i don’t think it’s blocking the current release) cc @enough-analyst-54434: https://github.com/pantsbuild/pants/issues/10455
e
I'll think about this more a bit later, but to check my understanding of reality, is it true that: There is no longer a build graph. A rule may choose to look at a field, any field, and interpret the contents of that field as some form of dependency it then operates against locally or via recursion. We happen to have the holdover single dependencies field, but would could have several or none on any given field set and we could even have a dependency expression field that is not a collection.
w
that has ~always been true, in both v1 and v2. but at the same time, we have tried to maintain the illusion of a build graph for the purposes of the graph introspection goals: 
./pants dependencies/dependees/list
etc.
e
Well, v1 had BuildGraph. OK. Well introspection goals are completely up for rewrite and deserve little consideration. We are fundmentally changing the nature of the metadata the metadata goals work on.
w
and that continues to be useful/necessary, afaict. we’ll need something that goes deeper at some point to allow people to scale up CI environments that can’t just run 
./pants test ::
by doing more precise change detection (but it’s unclear how far off that is with remoting)
e
But my point is change detection depends on the rule involved. A scala parser will need to follow dependencies on other scala and java files to handle the fallout of a changed file correctly. A scaladoc file will follow other fields, say if the scaladoc depends on some css files over there that have changed.
w
@enough-analyst-54434: yea, it’s fine to completely change the behavior of those goals right now, and that has in effect already happened because eg 
./pants dependencies
will render file addresses now as well as target addresses.
e
Great - so we agree those goals should not really feature in any arguments.
w
(which i think everyone is coming around on as a “good thing”)
@enough-analyst-54434: hm. not sure about that. they* (or something like them) need to exist, and be relatively consistent. they’re how people will “understand” inference
e
I fully disagree. They understand inference based on the source file they deal with. Scala source files have two independent inferences. There is the compilation which they understand via imports and there is the markdown which they understand via relative paths in links.
h
so we agree those goals should not really feature in any arguments.
The point I would push back is that those goals are very useful right now for debugging with users, and to give them insight into what Pants is doing. For example, before generated subtargets, you could use 
./pants list f.py
to see all the owning targets. I think we should be open to changing these goals, but that we should make sure the end effect still facilitates debugging and allows your every day Pants user (who is not on Slack) to have some insight into what Pants is doing.
w
@enough-analyst-54434: i expect that this was a point on the way to a larger point you were making?
e
Agreed. But my point is you can always make a new metadata goal that reveals the new metadata structure. We should absolutely forget about how those goals work today or yesterday in any arguments. Those arguments are noise.
w
a caveat is just timing. we don’t have time to invent replacements before alpha i don’t think. i can imagine replacements. but i expect that they will continue to exist at alpha, and 2.0 unless we get on top of everything and decide that it’s a priority.
h
But my point is you can always make a new metadata goal
Yeah, I think I agree. With an acknowledgment that we may be breaking some current workflows, e.g. scripts that use 
dependencies
. We’re already doing this by not having ported most goals like 
minimize
and 
paths
, and that’s the point of 2.0. But we should continue to keep it in mind where we are breaking things for v1 users.
e
@witty-crayon-22786 no - I just have held that typed targets in BUILD file should not exist in my head for a very long time now, only rule configuration should exist since thats all the rules need and all the user needs. As such I want to make sure when I look at all this in detail later if it looks like there is ratholing on a magic singular dependencies field in the base of every BUILD file target I can apply energy to thinking through that differently.
w
@enough-analyst-54434: got it. yea, i expect that we’re in agreement there. that’s mentioned in there as “targets are metadata applied to files” rather than “targets are a collection of files with some metadata”
it feels very right to me.
(the issue doesn’t discuss syntax changes, because i think they can follow independently if the model is solid.)
e
OK. To be clear I take “targets are metadata applied to files” a bit further than those words alone and think "“targets are metadata needed by various rules applied to files”. The implication there is there is no BUILD file target type, there is simply a collection of fields needed by different rules. If you add a new ruleset tomorrow that needs more metadata, you just add those metadata fields to the existing untyped target BUILD file.
w
@enough-analyst-54434: that part feels orthogonal. but maybe there is a connection i’m missing
e
The connection is the plural dependencies I led with. You can't really support that naturally without this.
w
ie, it seems like it deals more with “what syntax do i use to apply metadata”, rather than the fundamental question of whether the atomic unit is the file or the target
@enough-analyst-54434: plural or singular dependencies would still be metadata, i think.
e
It would. But my 1st gloss was the case of a single file you wanted to apply metadata two that contained two different types of dependencies used by two different dependency inference rule sets (compile vs doc gen is the running example). With typed targets in BUILD files if those two inference rule sets both need manual metadata we need to write down two targets in the BUILD file, each owning the 1 source file. At that point in in the gloss alarm bells went off re ownership which I know has figured into these discussions and implementations. As such, this seemed worthwhile clarifying for my later deep dive.
w
got it. yea, in case of multiple owners, where inference fails, there will need to be disambiguating syntax. there is general consensus around 
file@target
or something like it.
e
So I think the summary is target types in BUILD files are more than syntax since they can force un-necessary multiple ownership.
w
not sure about that. both sets of metadata existing was presumably necessary, unless the metadata ambiguity is a fight between two sets of rules (which is i think what you are implying, but i haven’t seen that yet in practice)
anyway, yea, sounds like we’re on the same page.
e
Ambiguity is not required. In today's model a single python source file may need to supply an 
entry_point
to a 
python_binary
target and a 
package
to a 
jvm_resources
target that supports prefixing resources with a package un-related to their location on disk. That requires two targets in the BUILD file today and multiple ownership ensues. With an untyped target metadata bag, you just add the two required fields for the two different rules to the one target and get no artifical ownership issues.
w
you also get less*** validation of your BUILD files… it’s not an open and shut win i think.
but yea. got it.
e
You get the same validation. Each field has individual validation and the rules in play for a given run validate they have the needed set of fields.
w
you would get less at parse time, because you’d lose the repetition of `python_library`and 
entry_point
needing to be in alignment. on the other hand, you lose the repetition, which is a benefit.
@enough-analyst-54434: but as mentioned above, i’m pretty sure that combining 
entry_point
and 
provides
or w/e other field has been much more rare in practice than combining 
entry_point
and 
entry_point
e
I think I agree.
Open in Slack
PreviousNext
Powered by