Hi I have a peculiar problem when building Docker images wit Pants #general

Hi! I have a peculiar problem when building Docker...

elegant-art-7479

05/06/2025, 3:31 PM

Hi! I have a peculiar problem when building Docker images with Pants, locally. When building an image, Docker wants to pull the base image, so

load metadata <image name>

appears in the logs. However, it hangs forever, there are no warnings or network activity during this. When I pull the base image manually, the build works fine. I doubt it's a network-related problem because we pull various dependencies during the build, and it works fine. Building with Docker directly also works fine. Any idea what's wrong here? I browsed Slack and all threads that contain a mention of

load metadata

show an error, while in my case there's no error, it just keeps hanging. CC @clean-policeman-96974

elegant-art-7479

05/06/2025, 3:41 PM

I tried running Pants with

-ldebug

and running the logged command, but it works

mammoth-area-83300

05/06/2025, 7:31 PM

to clarify, is

pants package

working now?

👎 1

elegant-florist-94385

05/06/2025, 9:20 PM

if you run

pants --keep-sandboxes=always package <docker image target>

you can check the output for a line describing where the sandbox is for that particular docker image. (probably something like

/tmp/pants-sandbox-21asdfec

) Then you can

cd

to that sandbox and inspect whatever dependencies pants inferred, and view the

__run.sh

script, which is the exact command pants uses to do that job (ie. to call docker). you can try running it yourself or tweaking it and running to debug (and then figure out how to map your edits back into pants itself)

👀 1

elegant-art-7479

05/07/2025, 5:58 AM

took the sandbox dir from this piece of log:

Copy code

07:48:22.20 [INFO] Preserving local process execution dir /private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-f4K0gD for Building docker imag
e us.gcr.io/fathom-containers/processing:test-arm

and saw:

Copy code

> pwd
/private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-f4K0gD

> tree
.
├── _binary_shims_a0fab74837aed5e4659100d5a62ba49444cd2d29cdb31edd0c2b536aa5f6210a -> /private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/immutable_inputscuHfa4/.tmp2oqBlA/a0fab74837aed5e4659100d5a62ba49444cd2d29cdb31edd0c2b536aa5f6210a
└── docker
    └── processing-test
        ├── Dockerfile_aarch64
        └── requirements.txt

4 directories, 2 files

there's no

__run.sh

script. The files inside

docker

are basically copies of the files from the repo, so they don't bring anything new

elegant-art-7479

05/07/2025, 6:00 AM

I do see

__run.sh

when looking at a sandbox for another target:

Copy code

07:48:21.62 [INFO] Preserving local process execution dir /private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-khgdSw for Test binary /usr/loc
al/bin/docker.

but it looks like it just smoke-tests a given binary:

Copy code

> pwd
/private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-khgdSw

> tree
.
└── __run.sh

1 directory, 1 file

> cat __run.sh 
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-khgdSw
env -i  /usr/local/bin/docker -v

> ./__run.sh 
Docker version 28.1.1, build 4eba377

elegant-art-7479

05/07/2025, 6:13 AM

I tried looking for a

__run.sh

in some other sandbox in the same temp dir that hosts the sandboxes, and found one related to running Docker! it was created a moment ago - is it possible that the file is created with some delay, from some Pants daemon (if there's any)?

elegant-art-7479

05/07/2025, 6:33 AM

using this

__run.sh

, so far I found out that when using the binary shims provided by Pants, the build hangs, but when forwarding my machine's

PATH

PATH="$PATH"

, it works fine. It looks like either an absence of some binary or a different binary makes Docker hang

elegant-art-7479

05/07/2025, 7:05 AM

I managed to narrow it down to

PATH="/usr/bin:/Users/piotr/google-cloud-sdk/bin"

where it still works (loads the metadata)

elegant-art-7479

05/07/2025, 8:03 AM

works with

/usr/bin

PATH

, at the beginning, and then the sandbox:

Copy code

PATH="/usr/bin:/private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-GkSJY0/_binary_shims_a0fab74837aed5e4659100d5a62ba49444cd2d29cdb31edd0c2b536aa5f6210a"

and works if it's at the end:

Copy code

PATH="/private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-GkSJY0/_binary_shims_a0fab74837aed5e4659100d5a62ba49444cd2d29cdb31edd0c2b536aa5f6210a:/usr/bin"

doesn't work with just the sandbox:

Copy code

PATH="/private/var/folders/p0/0f7r5rd53d9f41b7_c2rx7d40000gn/T/pants-sandbox-GkSJY0/_binary_shims_a0fab74837aed5e4659100d5a62ba49444cd2d29cdb31edd0c2b536aa5f6210a"

so the hypothesis is that the lack of access to some binary in

/usr/bin

makes some call hang. There are almost 1000 binaries in my

/usr/bin

, so once I add all of them to

pants.toml

, to

docker.tools

, with bisection it will be like a dozen of steps to narrow down to what extra we need to add 😄

elegant-art-7479

05/07/2025, 8:25 AM

it turned out that we were missing

basename

and

grep

in our tools to include in the sandbox and thus

PATH

, but I don't see any uses of it in the gcloud auth script. Anyway, it works!

elegant-art-7479

05/07/2025, 8:32 AM

• actionable from Pants' side: why

__run.sh

isn't always available • ideally we'd understand why a missing tool in

PATH

was causing the build to hang, e.g. by replacing

grep

with a script that logs something and/or fails hard and loud

elegant-art-7479

05/07/2025, 10:55 AM

I intercepted the calls to

grep

and despite not being able to get the parent process, I got the args (

"$@"

), these are:

-qxFe 3.9.22

. 3.9.22 is my current active version of Python, via pyenv. After this clue, it was easy to track down the call to `grep`: https://github.com/pyenv/pyenv/blob/bc7b6cb6b50c1bf86891b1a34c43203919fe7ed3/libexec/pyenv-latest#L46

9 Views

Open in Slack

Previous Next