regarding parameterization: Eric’s comment yesterday highlighted that i had not considered

overrides

while finalizing the syntax with

@

first, followed by

#

. after thinking it through, i agree, and have updated the design doc to swap those back, and to add a section on the topic

And we use the

:generator

snippet to locate where the generated target comes from. It also enforces disambiguation naturally Otherwise we need "address-less target generators" I think: https://docs.google.com/document/d/17XvhHlx8jBn4K9qFRZPe7qaKJQjTfiKMKYRubnzWxjA/edit. Which involves lots of discussion + implementation work

sortof: the alternative would be to eagerly run all generators at AddressFamily expansion time. which was in effect already what was happening for CAOFs

That eager expansion happens a lot more frequently now because of the change we made for specs to match generated targets

and because of inference. it will basically always happen.

This is interesting, we should 100% block 2.10 on resolving this imo. What's interesting to me about this is that the target generator will still be addressable, unlike the "address-less target generators" thing. That was my least favorite part of the proposal, why you can sometimes do a thing and sometimes can't

i suppose that it also might eventually be useful for

#

to mark “completely synthetic targets”, if we start creating targets from scratch.

i suppose that it also might eventually be useful for # to mark “completely synthetic targets”, if we start creating targets from scratch.

Meaning you want to reserve

#

for a different future use?

the same use, really:

#

marking a target that doesn’t actually exist in a BUILD file. so if you generated a “default” scala toolchain Target, for example, you’d generate it with a

#

address like

//#default-scala-toolchain

or something.

While I don't love the verbosity of

//:reqs#Django

rather than

//:Django

, I do think there's some value to being able to easily trace where something is coming from. For example,

debug_hint

with stacktraces. And then, again, avoids the problem of

//:reqs#Django

vs

//:poetry#Django

in the same repo. Which your design doc points out is an actual real issue, that macro authors etc need to reason about address disambiguation

also, i think that given its position in the address, the limitations on syntax inside of

#

aren’t really that bad… the only character you’d be forced to escape would be

@

That's okay, that's my reaction too! I don't love how complex addresses have become, but I also do really think it's justified / a necessary evil. Less offensive than the alternatives

Agree it’s a healthy reaction, and always good to take a moment to reevaluate also “known truths”.

…argh. i’m spinning on this. looking more into the implementation of supporting

overrides={'t.py': {"resolve": parameterize(..)}

, there are challenges: https://docs.google.com/document/d/1mzZWnXiE6OkgMH_Dm7WeA3ck9veusQmr_c6GWPb_tsQ/edit?disco=AAAAVIAE4Ds

making target generators “not real targets”… i.e., not giving them fields at all

I've roughly proposed that in the past https://github.com/pantsbuild/pants/issues/13086, but we can't to keep the alias mechanism +

./pants peek

yea… and your “generators are all macros” proposal, which i can’t find anymore.

But yeah what makes sense in my head is that you're not actually parametrizing the generator target...you're really parametrizing the generated targets. And overrides would override that.

python_sources(
   resolve=parametrize("a", "b")
   overrides={
      "utils.py": {"resolve": "c"},
      "another.py": {"resolve": parametrize("c", "d")},
},

should result in:

dir/f.py@resolve=a
dir/f.py@resolve=b
dir/utils.py
dir/another.py@resolve=c
dir/another.py@resolve=d

that only works if the target generator has no fields itself: otherwise there is no clear choice of field for it.

Gah I do wish target generators weren't targets. My main insight is what Pants actually cares about are atom targets. We should do everything to make those atom targets as precise as possible. Everything else is only for boilerplate reduction

removing their fields would be a step in that direction, i suppose.

What would that look like?

you would still declare the Fields on the generator type, but they would never be passed to the constructor.

Yes I think so! Can't remember if I put the comment that I suspect parametrization must leak into target generation `@rule`s, and that is okay

frankly… they are super redundant right now, so diving in on that is both exciting and scary.

I think that when you parametrize a target generator, you aren't actually creating multiple copies of the target generator. You're signaling instead that its generation rule should create multiple copies of each generated target (unless overridden)

yea. given the redundancy, that’s going to take some refactoring. but yea.

Target generators are already really weird, like how

tags

exists both on the generator itself and all generated targets. So I don't think the above proposed semantics will be that confusing. The docs try to push really hard that core message about atom targets being what matters

ok. regarding generator targets becoming less significant: (mostly) removing their fields seems like a step in that direction, which would align with optionally (in the future) not creating them at all if they don’t have a name

I'm open to considering that, although not sure it's necessary? I guess it would be if you parametrize one of its fields..I wonder if we could work around that another way, like literally showing

parametrize("a", "b")

in the

./pants peek

for it. (although JSON serialization is a thing)

I’m open to considering that, although not sure it’s necessary? I guess it would be if you parametrize one of its fields..I wonder if we could work around that another way, like literally showing 

parametrize("a", "b")

 in the 

./pants peek

 for it. (although JSON serialization is a thing)

that, and the Field itself must actually be constructed (with some value)

I do think it's helpful to use

./pants peek

on target generators because it lets you programmatically understand your BUILD files. But I don't think that is as crucial as

overrides

working as described above