hundreds-father-404
09/21/2022, 8:02 PMIn order to transparently support remote execution where possible (while falling back to other environments if necessary)
witty-crayon-22786
09/21/2022, 8:05 PMhundreds-father-404
09/21/2022, 9:24 PMenvironment
. Much less common is defining the environment. So, optimize for that 95% use case being simple
Rather than inventing a new DSL for the environment
field, like platform:linux_x86_64
and image:centos6
, add this to the environment target definitions:
docker_environment(prefer_local_environment=True)
remote_environment(fallback="centos6")
remote_environment(fallback="__local__") (the default)
remote_environment(fallback=None) (i.e. error)
Then consuming targets still set environment="centos6"
or environment="remote"
, while getting the benefits of fallbacksdocker_environment
or remote_environment
targets with different fallback behaviors. (They can use Python syntax in BUILD files or defaults for DRY)environment
field DSL is that it allows you to express "prefer RE, then fallback to local if a compatible platform, else Docker" (thanks to prefer_local_environment
)remote_environment
targets, maybe w/ a global default via options -- no need to update call siteswitty-crayon-22786
09/21/2022, 9:56 PMA small but neat benefit of this approach rather than thethat would be equivalent to defining three environments, and then matching them in order, i think?field DSL is that it allows you to express āprefer RE, then fallback to local if a compatible platform, else Dockerā (thanks toenvironment
)prefer_local_environment
hundreds-father-404
09/21/2022, 9:57 PMI think fortunately this design would be very easy to implementWoot, took a whole of 15 minutes to implement remote environment š https://github.com/pantsbuild/pants/pull/16955 docker should be easy to add before logging off tonight, if we like the design
witty-crayon-22786
09/21/2022, 9:58 PMbut speculation actually maybe makes it look more like a tree than a linked list? so maybe thatās an advantage?well⦠no, maybe not. you donāt need a tree⦠rather, two lists of environments to speculate with.
hundreds-father-404
09/21/2022, 9:59 PMthat would be equivalent to defining three environments, and then matching them in order, i think?You need a syntax to express "only Docker if you can't use local". So I don't think ordering that list as RE -> Local -> Docker would work, because local always is a match
witty-crayon-22786
09/21/2022, 10:00 PMYou need a syntax to express āonly Docker if you canāt use localā. So I donāt think ordering that list as RE -> Local -> Docker would work, because local always is a matchright: the syntax i had suggested in the Appendix was something like
platform: linux
hundreds-father-404
09/21/2022, 10:04 PMi think that this will workwhat does "this" mean here?
witty-crayon-22786
09/21/2022, 10:05 PMwhat does āthisā mean here?your design
hundreds-father-404
09/21/2022, 10:07 PMbecause each list head that you start walking from might send you somewhere differentI was thinking only adding
fallback_environment
to remote_environment
. so you could go RE -> RE -> RE -> Docker -> Local, theoretically, but that seems weird
For Docker, it doesn't seem necessary to fallback to another Docker env? And falling back to an RE env seems weird, that you'd prefer RE. Hence why I was thinking Keep It Simple and have prefer_local_environment: bool
witty-crayon-22786
09/21/2022, 10:08 PM@witty-crayon-22786 whatās the advantage of having each consumer use this new DSL, vs the approach in https://github.com/pantsbuild/pants/pull/16955?there may not be one. i think that the impact of your design is that the environment
name
becomes even more central, because it essentially becomes a bit like a strategy name: your centos6
environment might actually be defined as āremote centos6 if available, then local iff centos6, then centos6 docker imageāFor Docker, it doesnāt seem necessary to fallback to another Docker env? And falling back to an RE env seems weird, that youād prefer RE. Hence why I was thinking Keep It Simple and havewhere does that field go?prefer_local_environment: bool
hundreds-father-404
09/21/2022, 10:12 PMwhere does that field go?ah,
docker_environment(prefer_local_environment: bool)
, which means that if the local Platform == docker_environment(platform)
field, then use __local__
then local iff centos6This part won't be as powerful because I don't think we have enough information from a
docker_environment
target to match things like OS version. We only have the DockerPlatformField
+ Platform.create_for_localhost()
to go off of
We could add more fields to docker_environment
to make the matching more precise, if we found there was a need. Straw design, prefer_local_if_is_os=("centos", "6.*")
witty-crayon-22786
09/21/2022, 10:14 PMWe could add more fields tothat gets into the matching syntax⦠but matching inside the targets and giving them names might still be better from a user interface perspective ofto make the matching more precise, if we found there was a need. Straw design,docker_environment
prefer_local_if_is_os=("centos", "6.*")
environment=ā¦
being simple.hundreds-father-404
09/21/2022, 10:15 PMprefer_local_environment
for Docker, which meets the use case Jacob gave us. It defaults to False, and our help
will warn it solely matches by platform
I'm happy to name that field something conservative to mention the platform, so we can reserve more powerful matching in the futurewitty-crayon-22786
09/21/2022, 10:16 PMprefer_local
is thatās itās a bit odd to have the edge defined in the other direction: remote_environment
defines an outbound edge, this is an inbound edgelocal_environment
, and have that fall back to docker if need be?hundreds-father-404
09/21/2022, 10:21 PMname
from [environments].preview
. It doesn't really matter how the name centos6_or_local_linux
is defined, that's abstracted from consumers
What matters more is gently encouraging the environment authors to use good namesone thing with prefer_local is thatās itās a bit odd to have the edge defined in the other direction: remote_environment defines an outbound edge, this is an inbound edgeweird in what way? my head story is
using Docker is a niche feature, whereas local environment is the baseline. Have the niche config live on the Docker targetI'm not following also how it's different than
remote_environment(fallback_environment)
? It's the same general idea of "Maybe use this env, but fallback to X if possible or if required"witty-crayon-22786
09/21/2022, 10:26 PMItās the same general idea of āMaybe use this env, but fallback to X if possible or if requiredāitās the inverse: itās āuse that environment, but fall back to this one if not possibleā
hundreds-father-404
09/21/2022, 10:31 PMwitty-crayon-22786
09/21/2022, 10:43 PMlocal_environment
to grow more useful over time, the matching syntax you described (prefer_local_if_is_os
, etc) would also need to be on the local_environment
: i.e.: use this local environment if centos6, else that local environmenthundreds-father-404
09/21/2022, 10:46 PMi think that for the local_environment to grow more useful over time, the matching syntax you described (prefer_local_if_is_os, etc) would also need to be on the local_environmentagreed. I think we could probably avoid that
prefer_local_if_is_os
as scope creep until a user needs it. but we should make sure the design can accomdate itwitty-crayon-22786
09/21/2022, 10:46 PMlocal_environment
, not on the docker_environment
hundreds-father-404
09/21/2022, 10:47 PMlocal_environment
isn't defined for your current platform. That means just use the subsystem options as a default
otherwise, if you define 1 local env for e.g. M1s, we'd be forcing you to define for everything else. which may not be necessaryer: to be clear: i was saying that that is a motivation to put the definition of the match on the local_environment, not on the docker_environmentah, yeah, we'd need to replace
compatible_platforms
on local_environment
with something more powerfulwitty-crayon-22786
09/21/2022, 10:53 PM___local___
environment is not currently necessarily the beginning of a chain: we start by matching one via the platform__local__
hundreds-father-404
09/21/2022, 10:56 PMlocal_environment(
compatible_platforms=["linux_x86_x64"],
fallback_if_not_valid="centos6",
)
python_tests(environment="linux_or_centos6_docker")
So if you had two local Linuxes and wanted to prefer one to the other, we'd need a different resolution for localWhat does that mean?
witty-crayon-22786
09/21/2022, 10:57 PM___local___
, and it's always an explicit name?hundreds-father-404
09/21/2022, 10:58 PM(or maybe we get rid of local, and it's always an explicit name?I think we need
__local__
to handle the user story macOS vs Linux Python interpreter config
https://docs.google.com/document/d/1vXRHWK7ZjAIp2BxWLwRYm1QOKDeXx02ONQWvXDloxkg/edit#heading=h.3mi2qi3bz335witty-crayon-22786
09/21/2022, 10:58 PM> So if you had two local Linuxes and wanted to prefer one to the other, weād need a different resolution for local
What does that mean?if you define a
local_environment
that matches centos6, and another local_environment
for if not centos6, then currently things would explodeI think we needthat could be handled by chaining from an explicit first nodeto handle the user story__local__
macOS vs Linux Python interpreter config
hundreds-father-404
09/21/2022, 10:59 PMlocal Linux x86 -> Docker centos6
?witty-crayon-22786
09/21/2022, 11:01 PMlocal_environment(name="default", platform="macOS", fallback_to="linux", ..)
local_environment(name="linux", platform="linux", ..)
ā¦and then your default environment would be default
It might make more sense to me tomorrow morning, but would you mind maybe sketching it out with arrows, like?local Linux x86 -> Docker centos6
local_environment(name="default", platform="linux", more_specifically="centos6", fallback_to="any_linux")
local_environment(name="any_linux", platform="linux")
hundreds-father-404
09/21/2022, 11:02 PMwitty-crayon-22786
09/21/2022, 11:04 PM