dark horse here is also local process execution caching -- if we know a process execution is splittable we can cache it at a finer granularity

Except it also makes caching harder to reliably do…

hence formalizing "splittable"

That’s putting an awful lot of trust in rule-authors to get their caching semantics correct…

it can certainly be done manually... not sure it's worth making generic

no it is not trusting anyone it is making an interface to execute processes which allows us to uphold these guarantees

Unless done super carefully, it completely breaks caching…

users would not use the

SplittablyCacheableExecuteProcessRequest

unless this is the logic that applies to their use case

@average-vr-56795: as long as you partition deterministically, it shouldn't break caching

What if

x

and

y

aren’t truly independent? Or what if you incorrectly decompose the output tree so that running

x

is actually necessary to produce the output of running

y

, and in testing they happen to run together (or in the right order, or whatever)?

deterministically shouldn't be required

@witty-crayon-22786 Sure, but @aloof-angle-91616 is talking about dynamically partitioning…

but @aloof-angle-91616: splitting will primarily be beneficial when the overhead of each new task is low... so without nailgun/native-image, it would likely increase overhead

well then it would break and the rule author would realize their mistake?

It may only break a year later, when someone runs on a sufficiently fast or slow or loaded or whatever machine

@witty-crayon-22786 hence the graal PR which works and as noted here has a great first step towards doing this https://pantsbuild.slack.com/archives/C0D7TNJHL/p1549269379018000

@aloof-angle-91616: right. i'm just referring to priorities

i think the priority should be to make graal native-image work with zinc but nobody listens to me

Because people don’t fully understand the entire world, and I’m trying to guide people in ways that are hard to make mistakes with…

i don't see a set of circumstances that would allow us to ever introduce something that would make scalafmt go fast given the argument you are applying, and i would like to make scalafmt go fast (so perhaps we should recall that scalafmt is exactly the kind of thing that can be paralleled and cached per-file, recalling my message to @witty-crayon-22786 above). if your concern is that it is too general then we don't have to rush to shove it into

NailgunTask

. this is already going to be this separate

self.runjava_split()

method that nobody is using yet and i find it hard to see how it's easy to make mistakes here

I'm certainly happy to play with it in the context of something we know works well, but I think we should do that special cased in scalafmt, and think very very carefully about how, if at all, we generalise it

ok, this was my approach in the graal PR anyway. i do not have the experience to inform me of why this care is necessary. as i said when i introduced this, i am thinking of this "dynamic scheduling" (not referring to racing) as specifically the way we can start mixing and matching execution strategies (and more easily benchmarking them) to maximize rsc compilation speed, because there's not a clear winner right now, so thinking very carefully is fine, but i was interpreting that as saying enabling this sort of execution at all is scary and bad and that was confusing

Awesome 🙂 Sounds like we’re well aligned!