Which will require different, e.g., compilers or configuration

but not opaque to the planner right?

That's the trick: the planner doesn't have access to products. Sources are products!

The IDL could itself be codegen

Yeah - then that is not allowed today

you can only plan off of what you can touch today

Right

My claim is that this is somewhat reasonable, or at the very least it is a real world practicality we are going to have to address

And I'm starting to lean toward a planner model where the planner can decide that it cannot produce more of the plan graph until it gets access to its input products, perhaps some subset of them

I'd like an example for that one

I had a hell of a time with codegen and my example changed alot of code and notes I had

I thought I was going to have to give away alot ~1.5 weeks ago

But the example fleshed things out and in the end gave little away

ok - I'm going to desist and use the next hour for my last talk runthroughs

sgtm

I might pontificate here a bit or chat with others if they're around

I'm going to try to get some concrete examples on a whiteboard/in code

hmmmm

So I'm getting convinced that the proposed scheduler is doing what we want, but I am concerned about raising

ConflictingProducersError

when multiple plans can produce a product

Here's the concrete example:

We're attempting to compile a single target,

S

since it's a scala library

The product in question is

Sources.Of('.scala')

I claim that the

ThriftGen

task must satisfy this promise, but the product that results is empty

It is trivially cached, so I'm not concerned about perf, but if we're treating sources as products (as we should and must for codegen) and we're not explicitly typing targets (i.e. there's no difference between thrift_library and scala_library except configuration, and even then in simple cases the only difference might be the sources captured in the BUILD file), then during promise resolution time the

ThriftGen

task is obligated to look at an arbitrary target and say "Yeah, I can give you

'.scala'

files for this target, if you can give me

'.thrift'

files for it". Then a leaf on-disk-source-resolver task says "Yeah, I can give you

.thrift

files for this target. Here is an empty list, there are none."

Then during execution time

ThriftGen

says, "sure I can codegen zero thrift files, here is an empty output directory"

Then during execution of

ScalaCompile

, it gets some merged promises of

.scala

files, but in fact they only came from one task downstream, the one that resolves on-disk sources into a consumable product.

I suspect that here we're going to want to have a way to express how your promises can be satisfied in a more structured fasion

i.e., "Satisfy me this promise, and merge if multiple tasks satisfy it"

And perhaps even, "It's not an error if no one satisfies this promise"

And of course, "One and only one producer should satisfy this promise"