https://pantsbuild.org/ logo
#development
Title
# development
a

average-vr-56795

02/04/2019, 8:36 AM
@aloof-angle-91616 not sure what you mean by split/parallelizable tasks?
@aloof-angle-91616 nudge?
a

aloof-angle-91616

02/04/2019, 6:35 PM
that was something we would create
a

average-vr-56795

02/04/2019, 6:35 PM
Yeah, but I don’t know what it is… Could you describe it?
a

aloof-angle-91616

02/04/2019, 6:36 PM
i am typing
for things like scalafmt which can be invoked all at once with all files on the command line, or invoked in any number of processes partitioning the input files on their command lines, we can consider something like
Copy code
exe_req = SplitExecuteProcessesRequest(argv, input_digest, splittable_trailing_args)
actually that even be a function which generates an
argv
given some selection of input arguments
then when executing this the engine can choose to either execute it all at once, or split it into multiple `ExecuteProcessRequest`s
a

average-vr-56795

02/04/2019, 6:39 PM
Aha!
Sounds fun!
a

aloof-angle-91616

02/04/2019, 6:39 PM
we would need to sync
NailgunTask
with v2 execution a bit, but that doesn't need to involve anything more than some of the above logic in
NailgunTask
to start
ideally the parallelization part is generic for non-jvm tools, but i would also like for the engine/`NailgunTask` to choose between execution strategies, so not sure right now of the best path to take but i don't see it as that long of a path
a

average-vr-56795

02/04/2019, 6:46 PM
Yeah; I’d probably want us to get the hermetic stuff used a bit more before we start doing this, because optimising heuristics for strategies that are used and working is easier than optimising heuristics for strategies which don’t 🙂
But sounds exciting
a

aloof-angle-91616

02/04/2019, 6:53 PM
we could start by avoiding the engine at all then, but i think trying to phrase things in
NailgunTask
as `ExecuteProcessRequest`s (or some wrapper for nailgun invocations) is a good idea even if we don't execute them hermetically at all (i would love to use this as an excuse to make the subprocess executor do hermetic execution though, and that might not be that hard)
a

average-vr-56795

02/04/2019, 6:54 PM
The difficulty with making subprocess executor do hermetic is knowing what outputs it wants to materialise…
As for phrasing `NailgunTask`s as `ExecuteProcessRequest`s… We should probably work out if there’s going to be a hermetic nailgun story first, as that would probably shape the API quite a bit…
a

aloof-angle-91616

02/04/2019, 6:56 PM
i totally forgot
self.runjava()
is already generic to execution strategies
and the bit on knowing outputs to materialize also makes sense
i would be fine to start with introducing
Copy code
def runjava_split(self, argv_factory, main, classpath, splittable_args, ...):
 # ...
and making scalafmt our guinea pig, since there's an example already of that in the graal PR (although scalafmt is doing the splitting itself)
and
PartitionedTestRunnerTaskMixin
is a good place to look at because this is almost exactly the kind of logic it does (and i happened to edit it recently for go testing)