I have a dumb question about docker-image results caching. If I run

export GIT_COMMIT=$GIT_HASH
pants \
  --changed-since=origin/main \
  --changed-dependents=transitive \
  package publish

and my docker-images are defined like:

docker_image(
    name="docker",
    dependencies=[":docker_entry"],
    registries=["@aws_ecr"],
    repository="xyz",
    image_tags=["{build_args.GIT_COMMIT}"],
)

Will the change in image_tags (due to updated GIT_COMMIT) force all images to be republished, or only the ones that have code changes since

origin/main

?

Pants' interactions with the external docker daemon make the caching a bit awkward. Just focusing on packaging for now,

pants package some/docker:image

has a side-effect of making mutations in the external docker daemon (e.g. having an image of a given tag exist)... Even if an image exists in the pants cache, pants has no good way to know whether the external docker daemon has the correct state (for instance, someone might've run

docker system prune -f

to clear out everything if it was taking too much space). Thus, pants executes the

docker build

command for every docker image every time. (If the image already exists in docker, hopefully docker's internal cache makes those fast... But often that's not applicable on CI.)

Thank you for this context, it’s really helpful! On the second point (changed since) I think you’re right, I guess what I’m seeing is the first bit; on every new CI run, dockers internal cache is fresh, and I guess our Bazelremote instance (and pants local cache, which I believe is restored) are not holding any useful artifacts. Would using a remote build farm be the solution to this? Or perhaps using docker cache-to/cache-from settings, hmm

yeah, either of those might help; cache-to/cache-from might be the easiest