#22520 pants packaging a docker image using a kubernetes buildx driver (and no docker daemon) New discussion created by gdfast These are lessons from troubleshooting the issue discussed here: https://pantsbuild.slack.com/archives/C046T6T9U/p1751926012583459 ### tl;dr To use a kubernetes buildx driver • pass along

KUBERNETES_SERVICE_[HOST|PORT|PORT_HTTPS]

env_vars to the [docker] backend config of pants.toml • consider explicitly setting buildx driver and namespace in the same env_vars config, like

BUILDX_BUILDER=<your k8s buildx driver name>

,

BUILDKIT_NAMESPACE=<your namespace>

• on the

docker_image

target, set

output={"type": "registry"}

• don't run

pants publish

on that target because

pants package

will upload to the registry ### Premise The ability of pants to optimally build docker images with pex files is pretty compelling (see this blog). This is made even better with buildx support in the pants docker backend, which enables multi-platform docker image builds. Normally, with docker desktop installed,

pants package

and

pants publish

of a dockerimage target using buildx "just works".

package

will build the image and store it locally,

publish

can push it up to a registry. However, I recently hit an issue moving the pants commands that worked on my laptop to a (Gitlab) CI job had some tricky failures to overcome. ### More about the CI environment For our CI jobs, our workers run a Docker image that has the docker CLI installed. There have long been security concerns and complications running docker in docker (dind). For that reason, the team in charge of CI infrastructure at my company does not have a docker daemon running on the CI container (no socket). Instead, there's a kubernetes cluster of BuildKit workers available for building the images triggered by a

docker buildx build

command. On my laptop, I have the following build backend

% docker buildx ls 
NAME/NODE           DRIVER/ENDPOINT     STATUS    BUILDKIT   PLATFORMS
default             docker                                   
 \_ default          \_ default         running   v0.23.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)
desktop-linux*      docker                                   
 \_ desktop-linux    \_ desktop-linux   running   v0.23.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)

In CI, we instead have

docker buildx ls
NAME/NODE                     DRIVER/ENDPOINT                                                       STATUS    BUILDKIT   PLATFORMS
ci*                           kubernetes                                                                                 
 \_ buildkit-platform-amd64    \_ kubernetes:///ci?deployment=buildkit-platform-amd64&kubeconfig=   running   v0.21.1    linux/amd64 (+4), linux/386
 \_ buildkit-platform-arm64    \_ kubernetes:///ci?deployment=buildkit-platform-arm64&kubeconfig=   running   v0.21.1    linux/arm64, linux/arm (+2)
default                                                                                             error
Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Notice "default" status is "error" and that last line... there's no default docker daemon ### Overcoming

pants package

failure 1 – "cannot determine Kubernetes namespace" On buildx version v0.19.2 I got

ERROR: no valid drivers found: cannot determine Kubernetes namespace, specify manually: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

And on the older buildx v0.11.1 I got

ERROR: no valid drivers found: cannot determine Kubernetes namespace, specify manually: stat /root/.kube/config: no such file or directory

There's no kubeconfig file because the CI worker pod is in-cluster, which means it should get its namespace and cluster information from the kubernetes server. To get this to work for

docker buildx build

launched by pants, I had to pass the

KUBERNETES_SERVICE_HOST

and

KUBERNETES_SERVICE_PORT

environment vars to the docker backend's env_vars. With that info exposed, the

docker buildx build

executed by pants was able to pull the info it needed to use the

ci

buildx kubernetes driver. ### Overcoming

pants package

failure 2 - "failed to copy to tar" Next,

pants package <docker_image target>

almost succeeded in building the image, but failed with the following error:

#7 ERROR: failed to copy to tar: rpc error: code = Unknown desc = io: read/write on closed pipe

Inspecting with

pants -ldebug

, we see that pants is running the following command /usr/bin/docker buildx build \ --platform=linux/amd64 \ --output=type=docker \ --pull=False \ --file ...

type=docker

cannot work as an output type for remote docker builds when there's no local docker engine running that can save the image. The solution was to set

--output=type=registry

on the docker_image. ❗ This means there's no need for a

pants publish

step:

pants publish

pushes up an image from the local engine to a registry, but the output of

pants package

on the docker_image target already was that registry. ### Putting it all together (working in pants v2.27.0) pants.[ci.]toml [docker] use_buildx = true env_vars = [ "BUILDX_BUILDER=ci", "BUILDKIT_NAMESPACE=buildkit-platform", "DOCKER_CONFIG=%(homedir)s/.docker", "KUBERNETES_SERVICE_HOST", "KUBERNETES_SERVICE_PORT", "KUBERNETES_SERVICE_PORT_HTTPS", ... ] build_verbose = true BUILD target docker_image( output={"type": "registry"}, ... pantsbuild/pants