#18915 environments-preview feature not exchanging DOCKER_HOST Issue created by juftin Describe the bug Enabling

environments

breaks the docker build process in CI/CD and complains that the

DOCKER_HOST

is in the wrong place. * * * We have a

docker_image

that depends on a

pex_binary

BUILD

pex_binary(
    name = "bin",
    dependencies = [
        ":lib",
    ],
    execution_mode = "venv",
    include_tools = True,
    layout = "packed",
)

docker_image(
    name = "docker",
    dependencies = [
        ":docker_resources",
        ":bin",
    ],
    image_tags = ["{build_args.PANTS_DISTRIBUTION_VERSION}"],
)

Due to some wheel compatibility issues, we cannot build this

pex_binary

(and subsequent

docker_image

) on our Macs, it only works up in CI/CD (self-hosted GitHub Actions runners) where the machine is

linux_x86

. To resolve this issue for ourselves locally, we enabled the

environments

feature BUILD

local_environment(
  name="local_linux_x86",
  description="Localhost x86 Linux Environment",
  compatible_platforms=["linux_x86_64"],
  fallback_environment="docker_x86",
)

docker_environment(
  name="local_docker_x86",
  description="Dockerized x86 Linux Environment",
  platform="linux_x86_64",
  image="python:3.8.16",
)

And we attached the environment to the

pex_binary

BUILD

pex_binary(
    name = "bin",
    dependencies = [
        ":lib",
    ],
    execution_mode = "venv",
    include_tools = True,
    layout = "packed",
    environment = "local_linux_x86",
)

The idea here is that the

pex_binary

needs to be built on a

linux_x86_64

platform and when that platform isn't available locally it should just use the

docker_environment

. This solution is working for us locally, but when we run it up in CI/CD it no longer works. Here is the issue that we're seeing in CI/CD: Log Output

15:54:05.51 [DEBUG] Starting: Scheduling: Building docker image ***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-batch-job:16.0.0-beta.1|dkr.ecr.us-east-1.amazonaws.com/moz-batch-job:16.0.0-beta.1>
15:54:05.54 [DEBUG] Starting: acquire_command_runner_slot
15:54:19.81 [DEBUG] Completed: setup_sandbox
15:54:19.86 [DEBUG] spawned local process as Some(1133) for Process { argv: ["/usr/bin/docker", "build", "--pull=False", "--tag", "***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest|dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest>", "--tag", "***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:16.0.0-beta.1|dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:16.0.0-beta.1>", "--build-arg", "PANTS_DISTRIBUTION_VERSION", "--file", "services/moz-helper-services/Dockerfile", "."], env: {"PANTS_DISTRIBUTION_VERSION": "16.0.0-beta.1", "PATH": "/tmp/pants-sandbox-q6Zu6z/.shims/bin", "__UPSTREAM_IMAGE_IDS": ""}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<2c4455e78a6e491e19a07eca09e00a2fc9d026b6e428026680cb253cbbf4d541>, size_bytes: 264 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<79b1e31b408edaf0ecb646a2deaf492f0d62436d46c7e020659821ef142b3191>, size_bytes: 184 }, tree: "Some(..)" }, immutable_inputs: {RelativePath(".shims"): DirectoryDigest { digest: Digest { hash: Fingerprint<e6c1927a092ff6beaa1fa4f372248c975cb51869b1c28245cb0281ebc6816ecb>, size_bytes: 78 }, tree: "Some(..)" }}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Building docker image ***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest|dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest> +1 additional tag.", level: Info, append_only_caches: {}, jdk_home: None, platform: Linux_x86_64, cache_scope: PerSession, execution_strategy: Local, remote_cache_speculation_delay: 0ns }
15:54:19.89 [INFO] Completed: Building docker image ***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest|dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest> +1 additional tag.
15:54:19.89 [DEBUG] Completed: Scheduling: Building docker image ***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest|dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest> +1 additional tag.
15:54:19.89 [DEBUG] Completed: acquire_command_runner_slot
15:54:19.89 [DEBUG] Running Building docker image ***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-result-processor:16.0.0-beta.1|dkr.ecr.us-east-1.amazonaws.com/moz-result-processor:16.0.0-beta.1> under semaphore with concurrency id: 1, and concurrency: 1
15:54:19.89 [INFO] Starting: Building docker image ***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-result-processor:16.0.0-beta.1|dkr.ecr.us-east-1.amazonaws.com/moz-result-processor:16.0.0-beta.1>
15:54:19.89 [DEBUG] Starting: setup_sandbox
15:54:19.89 [WARN] Docker build failed for `docker_image` services/moz-helper-services:docker. The services/moz-helper-services/Dockerfile has `COPY` instructions for source files that may not have been found in the Docker build context.

However there are possible matches. Please review the following list of suggested renames:

  * services.moz-helper-services/bin.pex => services/moz-helper-services


15:54:19.90 [DEBUG] Completed: `publish` goal
15:54:19.90 [DEBUG] computed 1 nodes in 44.638043 seconds. there are 13299 total nodes.
15:54:19.90 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.core.goals.publish.run_publish
    `publish` goal
  in pants.core.goals.publish.package_for_publish
    ..
  in pants.core.goals.package.environment_aware_package
    ..
  in pants.backend.docker.goals.package_image.build_docker_image
    ..

Traceback (most recent call last):
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.1rc2_py38/lib/python3.8/site-packages/pants/engine/internals/selectors.py", line 593, in native_engine_generator_send
    res = func.send(arg)
  File "/home/runner/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.1rc2_py38/lib/python3.8/site-packages/pants/backend/docker/goals/package_image.py", line 309, in build_docker_image
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building docker image ***.<http://dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest|dkr.ecr.us-east-1.amazonaws.com/moz-helper-services:latest> +1 additional tag.' failed with exit code 1.
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

As far as I can tell, enabling the

environments

feature confuses

pants

about where the

DOCKER_HOST

should be (our GitHub runners have

DOCKER_HOST

set to

unix:///run/docker/docker.sock

instead of the default

unix:///var/run/docker.sock

). We pass this environment variable down in our

pants.toml

file though: pants.toml

[docker]
env_vars = [
  "DOCKER_CONFIG=%(homedir)s/.docker",
  "DOCKER_DEFAULT_PLATFORM=linux/amd64",
  "HOME",
  "USER",
  "PATH",
  # used by action-runner-controller dind
  "DOCKER_CERT_PATH",
  "DOCKER_HOST",
  "DOCKER_TLS_VERIFY",
]
tools = [
  "dirname",
  "readlink",
  "python3",
  # These may be necessary if using Pyenv-installed Python.
  "cut",
  "sed",
  "bash",
  "sh",
]
default_repository = "{directory}"
build_args = ["PANTS_DISTRIBUTION_VERSION"]

For solutions to this we've tried upgrading to

pants==2.15.1rc2

, removing the

local_environment

completely and just using the

docker_environment

, and also using

pants.ci.toml

to override which environment to use, and even hardcoding the

DOCKER_HOST

env var inside of

pants.ci.toml

. All of this worked within CI/CD when we specify no environments at all, but doesn't work … pantsbuild/pants