https://pantsbuild.org/ logo
Join Slack
Powered by
<@U6ZRNH0TC> not sure what you mean by split/paral...
# development
a
@aloof-angle-91616 not sure what you mean by split/parallelizable tasks?
@aloof-angle-91616 nudge?
a
that was something we would create
a
Yeah, but I don’t know what it is… Could you describe it?
a
i am typing
for things like scalafmt which can be invoked all at once with all files on the command line, or invoked in any number of processes partitioning the input files on their command lines, we can consider something like
Copy code
exe_req = SplitExecuteProcessesRequest(argv, input_digest, splittable_trailing_args)
actually that even be a function which generates an 
argv
given some selection of input arguments
then when executing this the engine can choose to either execute it all at once, or split it into multiple `ExecuteProcessRequest`s
a
Aha!
Sounds fun!
a
we would need to sync 
NailgunTask
with v2 execution a bit, but that doesn't need to involve anything more than some of the above logic in 
NailgunTask
to start
ideally the parallelization part is generic for non-jvm tools, but i would also like for the engine/`NailgunTask` to choose between execution strategies, so not sure right now of the best path to take but i don't see it as that long of a path
a
Yeah; I’d probably want us to get the hermetic stuff used a bit more before we start doing this, because optimising heuristics for strategies that are used and working is easier than optimising heuristics for strategies which don’t 🙂
But sounds exciting
a
we could start by avoiding the engine at all then, but i think trying to phrase things in 
NailgunTask
as `ExecuteProcessRequest`s (or some wrapper for nailgun invocations) is a good idea even if we don't execute them hermetically at all (i would love to use this as an excuse to make the subprocess executor do hermetic execution though, and that might not be that hard)
a
The difficulty with making subprocess executor do hermetic is knowing what outputs it wants to materialise…
As for phrasing `NailgunTask`s as `ExecuteProcessRequest`s… We should probably work out if there’s going to be a hermetic nailgun story first, as that would probably shape the API quite a bit…
a
i totally forgot 
self.runjava()
is already generic to execution strategies
and the bit on knowing outputs to materialize also makes sense
i would be fine to start with introducing
Copy code
def runjava_split(self, argv_factory, main, classpath, splittable_args, ...):
 # ...
and making scalafmt our guinea pig, since there's an example already of that in the graal PR (although scalafmt is doing the splitting itself)
and 
PartitionedTestRunnerTaskMixin
is a good place to look at because this is almost exactly the kind of logic it does (and i happened to edit it recently for go testing)
Open in Slack
PreviousNext
Powered by