#23411 docker_image builds fail with "docker-credential-osxkeychain not found in $PATH" after macOS tmp reaper deletes pantsd's materialized immutable_inputs shims Issue created by riisi AI disclaimer This investigation performed with Claude (Fable 5). I've been noticing this issue periodically in the past and I think i've also worked around by killing pantsd, or meddling with pants.toml docker tool options which I think may have the same effect (cache busting). Describe the bug On macOS, a long-running

pantsd

(3+ days) starts failing all

docker_image

builds with a misleading Docker credential error:

#2 [internal] load metadata for <http://docker.io/library/python:3.13-slim|docker.io/library/python:3.13-slim>
#2 ERROR: error getting credentials - err: exec: "docker-credential-osxkeychain": executable file not found in $PATH, out: ``

The root cause is that Pants materializes the

[docker].tools

binary shims under

$TMPDIR/immutable_inputs*/

, macOS's periodic temp cleaner deletes files in

/var/folders/.../T/

not accessed for ~3 days, and pantsd memoizes "already materialized" without re-verifying the directory exists. Subsequent docker builds run with

PATH

pointing at a deleted directory. The error is accompanied by the unrelated/red-herring warning "The Dockerfile has

COPY

instructions for source files that may not have been found in the Docker build context … suggested renames: . => app.py" (printed heuristically on any failed docker build), which sends debugging in the wrong direction. The credential helper is installed and works fine from a normal shell. Nothing in the repo, Docker Desktop, or shell environment changed; the same target built successfully days earlier. Fresh shells don't help; restarting pantsd does. Root cause analysis 1. Pants invokes

docker build

with

env -i ... PATH=<sandbox>/_binary_shims_<digest> -> $TMPDIR/immutable_inputsXXXX/<digest>

(visible via

--keep-sandboxes=always

→

__run.sh

). The shim dir contains

#!/bin/bash

/

exec "<abs path>" "$@"

scripts for each

[docker].tools

entry. 2. pantsd in this case had been running for ~6 days. macOS's periodic maintenance reaps files under

/var/folders/.../T/

not accessed in ~3 days. The daemon's

immutable_inputs*

directories were verifiably gone from

$TMPDIR

while pantsd kept running. 3. pantsd does not detect the deletion. Tool discovery still succeeds (a missing helper at discovery time produces a hard

BinaryNotFoundError

instead — verified, ruling that out), but the materialized shim dir referenced in the process

PATH

no longer exists.

ImmutableInputs::path_for_dir

memoizes materialized digests in an in-memory

HashMap<Digest, OnceCell<PathBuf>>

(

src/rust/fs/store/src/immutable_inputs.rs

) and returns the cached path without re-checking it exists on disk. 4.

docker

itself still executes because Pants invokes it by absolute path (

/usr/local/bin/docker

); only the

PATH

-resolved credential helper lookup inside buildx fails — producing a Docker-shaped error for a Pants-state problem. Confirmation: killing pantsd (or anything that forces a scheduler re-init, e.g. changing a daemon-level option like

--keep-sandboxes

) re-materializes the shims and the identical build immediately succeeds. Reproduction sketch 1. On macOS with pantsd running and a

docker_image

target whose base image pull consults a credential helper (

credsStore

in

~/.docker/config.json

), build once (succeeds). 2. Delete

$TMPDIR/immutable_inputs*

while pantsd stays alive (simulating the 3-day reaper). 3. Build again → the credential error above. Workaround pkill -f 'pantsd \[<buildroot>\]' # then re-run the build or set

local_execution_root_dir

to a location the OS reaper doesn't touch. Suggested fix Validate that memoized

immutable_inputs

materializations still exist on disk before reuse (or re-materialize on miss), similar in spirit to the named-caches staleness reports in #21973 / #22098. Even just re-statting the digest dir per session would prevent days-old daemons from handing out dangling

PATH

entries. Pants version 2.25.0 (the memoization in

<http://immutable_inputs.rs|immutable_inputs.rs>

is unchanged on

main

, so this should affect all recent versions) OS macOS (Darwin 25.5.0), Docker Desktop 4.70.0 (CLI 29.4.0),

credsStore: osxkeychain

Additional info

pants.toml

[docker].tools

includes

docker-credential-osxkeychain

,

docker-credential-desktop

, `docker-credential-gcloud`; all present on

PATH

at discovery time. pantsbuild/pants