aloof-angle-9161606/28/2019, 6:19 PM
as a root input
(2) pants resolves the appropriate binary tools for the linux platform (this already works)
(3) pants constructs the snapshot for linux from those tools (this also already works, it's just not made into a snapshot because the tasks are still v1, because i was waiting for local process execution caching)
(4) pants sends over the snapshot to the remote execution backend and receives the output (remote execution is known to work)
this could be done without introducing a
, and something like speculation could be implemented by instead just injecting the appropriate
parameter into the rule subgraph that ends up triggering the execution (because we already have a rule subgraph that does platform-specific logic in native_toolchain.py). it's true we then need to have a way to communicate to rust whether to execute the process locally or remotely, so maybe either a field on
or a separate object might work for that, but the identity of the injected
param is exactly what distinguishes the appropriate subgraphs in the native backend right now. i'm concerned when i see the
in the doc because the native backend already does this exactly, without having to introduce new syntax or classes, or having to write code that knows anything about platforms except where it differs (which the native backend has already figured out).
i spent some time when building the self-bootstrapping native backend to represent the intricacies of what's actually used when people run a compile from the compilers their package manager installs so that it could work immediately on all supported platforms. that time was spent first building on @happy-kitchen-89482's work on
yield Get(PlatformSpecificBinary, ...)
to make it more injectable and extensible, and then adding lots of very basic descriptions of things like
in https://github.com/pantsbuild/pants/blob/847dbe46f4d64635c23d2146d36f2642f0bd7918/src/python/pants/backend/native/config/environment.py#L22 (and there's some moderately-complex but really-useful logic added in that file with
and decorators to make this all really composable).
i think that structure offers quite a lot to build off of. in the case of speculation, i don't think we need to have a "target" and "host" platform, because we can simply make sure we inject the appropriate
parameter at the start, as described above. i would really like the work we're doing with speculation to align with the existing logic. if we want a concrete example of platform-specific logic (since the jvm backend doesn't provide that), the native backend is very small and highly decoupled from other subsystems by design, and was written precisely to be injectable in this fashion.
i can't edit the doc and comments can be ignored, hence the slack message