https://pantsbuild.org/ logo
#general
Title
# general
g

gentle-painting-24549

05/03/2023, 5:23 PM
I’ve got a predicament with our local mac, and linux based CI/CD processes. We have a PEX artifact that gets packaged into a docker container. Due to some wheel compatibility issues, the PEX must be build for a linux x86 system. To get around this issue we started using environments (in particular a
docker_environment
) to build the PEX and subsequent
docker_image
target. Everything works great locally until it came time to publish our
docker_image
from CI/CD (GitHub Actions self-hosted runners). The build fails with the following error when I specify a
docker_environment
on the
pex_binary
target:
Copy code
Exception: Failed to obtain version from local Docker: error trying to connect: No such file or directory (os error 2)
As best as I can tell the
DOCKER_HOST
environment variable isn’t getting passed down into the build process and it’s not finding the docker binary when trying to run
docker -v
. I’ve specified
DOCKER_HOST
and a number of other vars in the
docker.env_vars
table and have also tried supplying them to the
docker_environment
target itself. I’ve seen a few somewhat-related issues around this in the slack history, and so I’m wondering if there’s an issue with my pants config or if
docker_environment
just isn’t ready for primetime. (I’m on pants 2.15.0, but have tried other versions too)
w

witty-crayon-22786

05/03/2023, 5:35 PM
i believe that
2.15.1rc2
fixes this.
👀 1
g

gentle-painting-24549

05/03/2023, 5:41 PM
👏 oh nice, I’ll give this a shot right now. Thank you @witty-crayon-22786.
Actually, I tested with
2.15.1rc2
yesterday and still ran into that error. I’m also not seeing any mention of
DOCKER_HOST
in the debug logs either.
Copy code
18:12:17.69 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.core.goals.publish.run_publish
    `publish` goal
  in pants.core.goals.publish.package_for_publish
    ..
  in pants.core.goals.package.environment_aware_package
    ..
  in pants.backend.docker.goals.package_image.build_docker_image
    ..
  in pants.backend.docker.util_rules.docker_build_context.create_docker_build_context
    ..
  in pants.core.goals.package.environment_aware_package
    ..
  in pants.backend.python.goals.package_pex_binary.package_pex_binary
    ..
  in pants.backend.python.util_rules.pex.create_pex
    ..
  in pants.backend.python.util_rules.pex_from_targets.create_pex_from_targets
    ..
  in pants.backend.python.util_rules.python_sources.prepare_python_sources
    ..
  in pants.core.util_rules.source_files.determine_source_files
    Get all relevant source files - environment:docker_x86
  in pants.engine.internals.graph.hydrate_sources
    Hydrate the `sources` field - (environment:docker_x86, packages/moz-service-proto/moz_service_proto/central_point.proto:../proto)
  in pants.backend.codegen.protobuf.python.rules.generate_python_from_protobuf
    Generate Python from Protobuf - environment:docker_x86
  in pants.backend.python.util_rules.pex_environment.find_pex_python
    Prepare environment for running PEXes - environment:docker_x86
  in pants.core.util_rules.subprocess_environment.get_subprocess_environment
    ..
  in pants.engine.internals.platform_rules.environment_vars_subset
    ..
  in pants.engine.internals.platform_rules.complete_environment_vars
    ..
  in pants.engine.process.fallible_to_exec_result_or_raise
    ..
  in pants.core.util_rules.environments.extract_process_config_from_environment
    ..

Traceback (no traceback):
  <pants native internals>
Exception: Failed to obtain version from local Docker: error trying to connect: No such file or directory (os error 2)
w

witty-crayon-22786

05/03/2023, 6:17 PM
what is
DOCKER_HOST
on the relevant machine?
g

gentle-painting-24549

05/03/2023, 6:18 PM
<tcp://localhost:2376>
w

witty-crayon-22786

05/03/2023, 6:19 PM
got it. it looks like we might only support
unix://
domain sockets currently
g

gentle-painting-24549

05/03/2023, 6:20 PM
And that’s via the
docker_environment
, right? Everything works as expected before adding the
docker_environment
(except we can’t build PEX/Docker images on our macs)
w

witty-crayon-22786

05/03/2023, 6:21 PM
yes. the
docker_environment
implementation uses a native client library rather than forking the
docker
cli
g

gentle-painting-24549

05/03/2023, 6:24 PM
Gotcha, that makes sense, although kind of frustrating. We’re using docker-in-docker inside of GHA which is likely why this is so complicated. The fix for our situation is probably some kind of logic that says to only mess around with
docker_environments
when running on our macs, and use the default behavior when inside of CI/CD)
w

witty-crayon-22786

05/03/2023, 6:27 PM
i’ll be honest: i don’t know the pros/cons of
tcp
vs
unix
sockets in this context. is that a choice that you made when setting up Docker here?
it looks like i could add a flag here (or attempt to do some automatic fallback), but some more context would help
mm… according to https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option , it also seems like because you’re using
2376
, an encrypted connection might be necessary
g

gentle-painting-24549

05/03/2023, 6:35 PM
I’m not exactly sure either. We’re using https://github.com/actions/actions-runner-controller/ for our self-hosted runners. Although we’re using an outdated version of it and it looks like they may have switched away from the TCP socket recently.
w

witty-crayon-22786

05/03/2023, 6:41 PM
ok. it looks like some very basic detection would fix the
unix
vs
tcp
aspect of this, but the TLS/SSL aspect is still unclear to me (it doesn’t get a unique address string, afaict).
g

gentle-painting-24549

05/03/2023, 6:52 PM
Thank you Stu!
Follow up here, I went through the process of upgrading our actions-runner-controller helm chart to the latest version, 0.18.0 -> 0.23.3, and the
DOCKER_HOST
variable is updated to
unix:///run/docker/docker.sock
inside our runners now. However, I’m still running into issues on
2.15.1rc2
. When I have both a
local_environment
and a
docker_environment
specified (only my
pex_binary
targets are specifically using this environment), I get this issue:
Copy code
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
When I remove the
local_environment
and leave only the
docker_environment
, I get the following issue:
Copy code
Engine traceback:
  in `publish` goal
  in Building local distributions - environment:docker_x86

ProcessExecutionFailure: Process 'Extract environment variables from the Docker image python:3.8.16' failed with exit code 126.
stdout:
OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd ("/pants-sandbox/pants-sandbox-nsMAG4") set in config.json failed: no such file or directory: unknown
w

witty-crayon-22786

05/04/2023, 7:53 PM
hm. are you sure that this is “docker in docker”, and not “docker from docker”?
if you’re connecting to a docker daemon outside of your container, the container won’t be able to expose files to other spawned containers
g

gentle-painting-24549

05/04/2023, 8:04 PM
🤔 I’m not sure about that - it’s likely docker-from-docker since we’re using K8S to run our GHA runners. I think my first solution was the right one where I had one
local_environment
(linux_x86) and one
docker_environment
. The
local_environment
falls back to the
docker_environment
so that CI/CD uses the
local_environment
to build the pex and our Mac’s use the
docker_environment
. The issue there is that the
DOCKER_HOST
environment variable doesn’t seem to get passed through so it’s looking for
unix:///var/run/docker.sock
instead of
unix:///run/docker/docker.sock
inside of CI/CD
All of this just worked inside of CI/CD before we tried to enable environments. Environments has unblocked us locally, but blocked us in CI/CD. Is there a way to just enable environments when running outside of CI/CD but disable them when inside of CI/CD via
pants.ci.toml
or something like that?
w

witty-crayon-22786

05/04/2023, 8:17 PM
Yes: there is an example of that in the docs.
g

gentle-painting-24549

05/04/2023, 8:20 PM
oh, I read through that early on but haven’t revisited it. I think that may be the magic solution here. Thanks Stu, I’ll give that a try here shortly.
w

witty-crayon-22786

05/04/2023, 8:22 PM
The
local_environment
falls back to the
docker_environment
so that CI/CD uses the
local_environment
to build the pex and our Mac’s use the
docker_environment
.
The issue there is that the
DOCKER_HOST
environment variable doesn’t seem to get passed through so it’s looking for
unix:///var/run/docker.sock
instead of
unix:///run/docker/docker.sock
inside of CI/CD
i’m not sure whether i understand the setup for this one: would you mind filing an issue about it?
g

gentle-painting-24549

05/04/2023, 8:23 PM
Yeah I can do that
Thanks for your help again Stu. I tried overriding which environment to use but still ran into the same issue. I raised an issue about it here: https://github.com/pantsbuild/pants/issues/18915