https://pantsbuild.org/ logo
#development
Title
# development
w

witty-crayon-22786

07/23/2020, 12:22 AM
after further reflection, i think the “tuple of file and a metadata source” model is quite clear, and that changes to the graph introspection goals are orthogonal-ish (or at least not an improvement to internal complexity). so: yay!
i do expect that one effect of (file, Target) will be that it no longer makes sense for the target object to “own” files after hydration
h

hundreds-father-404

07/23/2020, 12:26 AM
This is Benjy’s proposal, correct?
For now, it’s a tuple of (Address, optional file name) (Optional to accomadate targets with no source files, like
python_requirement_library
)
w

witty-crayon-22786

07/23/2020, 12:29 AM
mmm, true.
yea, what benjy said there
❤️ 1
h

hundreds-father-404

07/23/2020, 12:37 AM
Cool. I’m reverting generated subtargets right now
w

witty-crayon-22786

07/23/2020, 4:38 PM
so, i had a realization about this this morning
the effect of
(Target, Optional[File])
or
(Address, Optional[File])
is that it doesn’t address the fact that
Target
owns zero or more sources right now
i actually do think that
Address
being either a “for a file” or a “for a target” makes sense
h

hundreds-father-404

07/23/2020, 4:43 PM
is that it doesn’t address the fact that Target owns zero or more sources right now
It addresses in that you would have one pair for each file belonging to a target. If you don’t care about files, we strip those all back down into just the single `Address`/`Target`
w

witty-crayon-22786

07/23/2020, 4:44 PM
right, but i don’t think that actually gives you the type safety benefit
because you can use the inner thing accidentally
(…the same goes for Address itself, but with a lower surface area perhaps)
so, i think that how Benjy’s idea connected for me was “yes, it’s a tuple of file and metadata”
but one way to accomplish that is that
Target
itself
could have the optional file.
so a hydrated target would be basically
(Fields, Optional[Source])
which is roughly the same idea… it’s just that it’s actually target that is the thing that associates metadata with a source.
h

hundreds-father-404

07/23/2020, 4:48 PM
And what is
Address
? Is your proposal basically “generated subtargets, but always used no matter what”?
w

witty-crayon-22786

07/23/2020, 4:48 PM
and an Address continues to have a 1:1 correspondence with a Target: a file address gets you a Target owning the one file, but a target address would get you a Target owning no files, but depending on a bunch of file targets
h

hundreds-father-404

07/23/2020, 4:49 PM
I’m not following what the motivation there is?
w

witty-crayon-22786

07/23/2020, 4:49 PM
@hundreds-father-404: yea, basically? when you generate a base target, the effect is that it depends on the “sub/file” targets rather than owning the source
@hundreds-father-404: the motivation is that it is still almost exactly the “tuple of optional file and its metadata” approach
as in, i think this is the same idea, but with a different implementation
h

hundreds-father-404

07/23/2020, 4:51 PM
when you generate a base target
Do you know what BUILD files would look like? If we allow you to still have a target in a BUILD file that has multiple files in its
sources
, then I think this needs to be flipped. Our starting point is the base target which has multiple files. From there, we can generate the subtargets. I think you’re proposing that then from there, we can generate a new base target that no longer owns sources?
w

witty-crayon-22786

07/23/2020, 4:51 PM
but “a hydrated Target always has 1 or zero files” accomplishes the tuple idea in a different way that i think is very, very compatible with existing rules.
@hundreds-father-404: i’m not proposing BUILD file changes… this would be an implementation detail
an advantage of the base target depending on the subtargets is that it “just works” with all existing rules, i think.
depending on the base target is an alias for depending on “all of the files”
h

hundreds-father-404

07/23/2020, 4:54 PM
Okay, then yes, it sounds like we would have the original base target -> generated subtargets -> generated base target. Right now, we have original base target -> sometimes generated subtargets -- I’m not following why we need that new last step? What is the benefit? Generated subtargets already don’t require any changes to rules, minus
dependees
. My criticisms of generated subtargets are almost entirely based on the user experience of things like file args vs. target args giving a different experience. Afaict, this would still be the case.
w

witty-crayon-22786

07/23/2020, 4:55 PM
My criticisms of generated subtargets are almost entirely based on the user experience of things like file args vs. target args giving a different experience. Afaict, this would still be the case.
that part isn’t clear to me actually. it depends what the rules do with the targets.
in particular, if you have an in memory base target which owns no sources, and depends on a bunch of subtargets, what rules would do today would be run each subtarget individually
h

hundreds-father-404

07/23/2020, 4:56 PM
How so? With the current implementation, rules get Targets. They are agnostic to how those were given to it. They could have been explicit targets at 1-1-1 precision; explicit targets at 1 target to 1 file precision; or generated subtargets. The rule doesn’t need to care. It can if it wants to, but doesn’t need to.
w

witty-crayon-22786

07/23/2020, 4:56 PM
^ see my last
h

hundreds-father-404

07/23/2020, 4:57 PM
what rules would do today would be run each subtarget individually
This is already the case with generated subtargets, though? I don’t understand how this improves upon the current implementation.
w

witty-crayon-22786

07/23/2020, 4:57 PM
ie, you’d get some uniformity because a rule would always run at the file level, unless it wanted to do something else
@hundreds-father-404: i mean that it would happen uniformly, regardless of the CLI specs
because you would always generate the subtargets, and the base target would always depend on them rather than owning its own sources.
h

hundreds-father-404

07/23/2020, 4:58 PM
i mean that it would happen uniformly, regardless of the CLI specs
So, with something like the current
pytest_runner.py
implementation, you’re saying that no matter what we would always run one-file-at-a-time?
w

witty-crayon-22786

07/23/2020, 4:59 PM
with the current implementation, yea.
(so… it would be uniform… for better or worse.)
but i sortof think that that might be the natural consequence of Benjy’s tuple idea
… the rule would have to choose partitions/groups if it wanted them
(which at a basic level might be “group by base target” or something)
cc @happy-kitchen-89482: whole thread
h

hundreds-father-404

07/23/2020, 5:02 PM
it would be uniform
I see. And I acknowledge that uniformity is one of my main criticisms of generated subtargets / file level deps. An alternative, maybe simpler, approach would be to make changes so that file arguments always no matter what use generated subtargets. We could do that if we add the way to disambiguate between multiple owners, e.g.
@
. Then, with file args, you always get generated subtargets. With target args, you always get normal targets
w

witty-crayon-22786

07/23/2020, 5:03 PM
An alternative, maybe simpler, approach would be to make changes so that file arguments always no matter what use generated subtargets.
yes, i think that this should happen… this was one of the things that i think we agreed on in the design session
ie that
./pants test ::
and
./pants test '**/*'
would be different, and that that is ok.
h

hundreds-father-404

07/23/2020, 5:03 PM
There are I think two places I find the lack of uniformity confusing: - file args vs. address args give you a different experience - file args sometimes use generated subtargets, but sometimes don’t I think that second part is the more offensive to me.
w

witty-crayon-22786

07/23/2020, 5:04 PM
agreed as well.
two is more confusing. i can also see benefits to fixing the first one… what benjy has said about letting rules partition iff they care… sortof makes sense.
h

hundreds-father-404

07/23/2020, 5:05 PM
this was one of the things that i think we agreed on in the design session
Indeed. But we punted on it to avoid inventing this new disambiguation syntax. I think the past two weeks show it may be worth it.
👍 1
Let’s solve this in another thread maybe, but the other major blocker for me is the problem of how we know whether you depend on sibling files in the original base target.
w

witty-crayon-22786

07/23/2020, 5:07 PM
oh. sorry. was misunderstanding the question.
yea, elsewhere is fine.
h

happy-kitchen-89482

07/23/2020, 5:42 PM
Still reading through this, but it sounds like you're proposing that
target_type(sources=[a, b, c], field1=foo, field2=bar)
acts like a macro that expands to
Copy code
target_type(source=a, field1=foo, field2=bar)
target_type(source=b, field1=foo, field2=bar)
target_type(source=c, field1=foo, field2=bar)
?
More or less, anyway
w

witty-crayon-22786

07/23/2020, 5:43 PM
yes.
oh, with one very important tweak
it expands to
Copy code
target_type(dependencies=['a', 'b', 'c'], field1=foo, field2=bar)
target_type(source=a, field1=foo, field2=bar)
target_type(source=b, field1=foo, field2=bar)
target_type(source=c, field1=foo, field2=bar)
ie, the base target “depends on its files, but does not own any” and “each target owns zero or one file”
h

happy-kitchen-89482

07/23/2020, 5:51 PM
So why do we need the base target at all at this point (after we've expanded it)?
w

witty-crayon-22786

07/23/2020, 5:57 PM
@happy-kitchen-89482: because it is possible to depend on the base target. it has an address, and address/target should be 1:1