understood.

And we should work toward a better solution in the current engine

Okay, I'm going to claim that this is the right way to go: what ivy invalidates and truly operates over is a subgraph, BUT it has the ability to throw away a lot of the structure of that subgraph (it already does that in fact, its FingerprintStrategy was motivated by the need to not worry about JvmTarget hashes in thesubgraph). Now, when you actually construct a classpath in practice, it's also always for a subgraph, right? So really instead of requesting a classpath for a target, we should be constructing a classpath for the full subgraph we're operating over

I think it’s going to be hard to avoid having a context-sensitive execution model

However, that then propagates this issue further downstream

I think we should avoid context-sensitivity at all costs here. Including the cost of having Ivy work the way it does today

IMO the solution might be to write our own resolver that we can reason carefully about and prove works in a way that is amenable to this system

i think i agree.

(with patrick)

But the inability to compute cache keys for things at the edge of the graph because we don't know what their product inputs are is a deal breaker

It totally hoses the biggest feature of the new engine

yea.

if you pre-computed the transitive ivy dep graph and dropped it in BUILD files (or somewhere else), things would work.

which i believe is similar to what you guys have done already

Code-genning BUILD files using the results of Ivy was our first idea that eventually lead to pom-resolve

And seems like a totally valid path

but the difficulty in our situation is that we have many permutations, and we believe that that is a feature

I agree, but it's a feature that greatly increases the scope of the problem

yes

It could also be that we propagate this "resolve by subgraph" logic to its logical extreme and figure out some way to cleverly run ivy very fast and cache the results well

Not too different than how we've really adopted zinc

@bored-art-40741: is there a middle ground, where we can indicate that some tasks hash products, and some don't?

or do we really have to reify the entire thing to disk

That's kind of what I was thinking earlier. For jvm compile in particular, we'd want to turn off that feature using an option

Though in the new engine I think the interface itself just won't allow it

Products won't be available during the lifecycle method where cache keys are computed

unless we want to be more pragmatic about this =P

I think the pragmatism here will come from changing the way that we resolve rather than bending the entire engine design so we can scoot resolution through a loophole

As you said, 3rdparty/BUILD doesn't change too much, and if you change it it's okay to charge a big re-resolution penalty

At which point, keying all accessing to

compile_classpath

by subgraph instead of by target actually doesn't thrash the cache much