https://pantsbuild.org/ logo
Join Slack
Powered by
hi :wave: I have this very basic setup for docker...
# general
s
hi 👋 I have this very basic setup for docker
Copy code
# infra/base-images/images/chemistry/BUILD
files(
    name="deps",
    sources=[
        "requirements.txt", 
    ],
)

docker_image(
    name="chemistry",
    dependencies=[
        ":deps",
        "//infra/base-images/images/base:base",
    ],
    extra_build_args = [
        "baseimage=//infra/base-images/images/base:base",
    ],
)
and I use a default context
Copy code
use_buildx = true
default_context_root = "./"
When I run a "pants publish", I get the following error message
Copy code
09:36:38.63 [WARN] Docker build failed for `docker_image` infra/base-images/images/chemistry:chemistry. There are unreachable files in these directories, excluded from the build context due to `context_root` being 'infra/base-images/images/chemistry':

  * infra.base-images.images.base

Suggested `context_root` setting is '' in order to include all files in the build context, otherwise relocate the files to be part of the current `context_root` 'infra/base-images/images/chemistry'.
09:36:38.63 [ERROR] 1 Exception encountered:

Engine traceback:
  in `publish` goal

ProcessExecutionFailure: Process 'Building docker image europe-docker.pkg.dev/xxxxx/platform/chemistry/chemistry:ademaria-fix-fix-ci-pants-tags +2 additional tags.' failed with exit code 1.
stdout:

stderr:
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile:
#1 transferring dockerfile: 678B done
#1 WARN: InvalidDefaultArgInFrom: Default value for ARG ${baseimage} results in empty or invalid base image name (line 3)
#1 DONE 0.0s

#2 [internal] load metadata for europe-docker.pkg.dev/xxxx/platform/base/base:ademaria-fix-fix-ci-pants-tags
#2 ERROR: europe-docker.pkg.dev/xxxx/platform/base/base:ademaria-fix-fix-ci-pants-tags: not found
------
 > [internal] load metadata for europe-docker.pkg.dev/xxxx/platform/base/base:ademaria-fix-fix-ci-pants-tags:
------
Dockerfile:3
--------------------
   1 |     ARG basetag
   2 |     ARG baseimage=eu.gcr.io/xxxx/cuspai/base-images/base:${basetag}
   3 | >>> FROM ${baseimage}
   4 |     
   5 |     # Add fortran
--------------------
ERROR: failed to solve: europe-docker.pkg.dev/xxxx/platform/base/base:ademaria-fix-fix-ci-pants-tags: failed to resolve source metadata for europe-docker.pkg.dev/xxxx/platform/base/base:ademaria-fix-fix-ci-pants-tags: europe-docker.pkg.dev/xxxxx/platform/base/base:ademaria-fix-fix-ci-pants-tags: not found
✅ 1
Question, does the base image dependency need to be in the context?
correction: this build is running on the local docker I have removed from the post references to the remote buildkitd being used
b
Sorry for the trouble! Can you share the dockerfile?
❤️ 1
s
Copy code
ARG basetag
ARG baseimage=<http://eu.gcr.io/xxxxx/base-images/base:${basetag}|eu.gcr.io/xxxxx/base-images/base:${basetag}>
FROM ${baseimage}

yada yada
I stopped testing but I think extending the context to be the full repo, it seems to work.
will test later again
b
Ah okay, if you’ve solved it, that’s good, but let us know if not!
s
sure, thank you meanwhile!
Hi, unfortunately I do need your help! I have a pretty simple setup so I don't understand what's not working the ml image depends on 
infra/base-images/images/base:base
Copy code
pants dependencies infra/base-images/images/ml:ml
infra/base-images/images/base:base
infra/base-images/images/ml/mamba_environment.yaml:deps
infra/base-images/images/ml/pip_requirements.txt:deps
infra/base-images/images/ml/pip_requirements.txt:reqs
the BUILD is pretty straightforward
Copy code
files(
    name="deps",
    sources=[
        "pip_requirements.txt", 
        "mamba_environment.yaml"
    ],
)

docker_image(
    name="ml",
    dependencies=[
        ":deps",
        "//infra/base-images/images/base:base",
    ],
)
The dockerfile is essentially
Copy code
FROM infra/base-images/images/base:base
I when I run 
package
I get (on mac, using buildx)
Copy code
#22 writing cache manifest sha256:32a1aafbface55ae5fff05a68b43721604124b196be1c67cb869c9981011c984 0.1s done
#22 DONE 5.2s
------
 > importing cache manifest from europe-docker.pkg.dev/xxxxxxxx/platform/base/base:main}:
------

17:02:04.83 [INFO] Completed: Building docker image europe-docker.pkg.dev/xxxxxxxx/platform/ml/ml:ademaria-fix-fix-pants-build-tags +2 additional tags.
17:02:04.83 [WARN] Docker build failed for `docker_image` infra/base-images/images/ml:ml. There are files in the Docker build context that were not referenced by any `COPY` instruction (this is not an error):

  * infra.base-images.images.base/base.docker-info.json


17:02:04.83 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building docker image europe-docker.pkg.dev/xxxxxxxx/platform/ml/ml:ademaria-fix-fix-pants-build-tags +2 additional tags.' failed with exit code 1.
stdout:

stderr:
#0 building with "desktop-linux" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 514B done
#1 DONE 0.0s

#2 [internal] load metadata for <http://docker.io/infra/base-images/images/base:base|docker.io/infra/base-images/images/base:base>
#2 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for <http://docker.io/infra/base-images/images/base:base|docker.io/infra/base-images/images/base:base>:
------
Dockerfile:1
--------------------
   1 | >>> FROM infra/base-images/images/base:base
   2 |     
   3 |     # Add additional micromamba dependencies
--------------------
ERROR: failed to solve: infra/base-images/images/base:base: failed to resolve source metadata for <http://docker.io/infra/base-images/images/base:base|docker.io/infra/base-images/images/base:base>: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
r
This looks like a local image. If that's the case, pants doesn't support yet using a local image.
Copy code
infra/base-images/images/base:base
s
what do you mean by a "local image"? How is it different than https://www.pantsbuild.org/dev/docs/docker
r
I just saw that's your target address not the image URI. Ignore what I wrote
👍 1
s
it got me scared 🙂
hello @broad-processor-92400, I am still having issues with this. For some reason, referring to a docker image as a base doesn't seem to work
Copy code
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 677B done
#1 WARN: InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)
#1 DONE 0.0s

#2 [internal] load metadata for <http://docker.io/infra/base-images/images/base:base|docker.io/infra/base-images/images/base:base>
#2 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for <http://docker.io/infra/base-images/images/base:base|docker.io/infra/base-images/images/base:base>:
------

 1 warning found (use docker --debug to expand):
 - InvalidDefaultArgInFrom: Default value for ARG ${BASE_IMAGE} results in empty or invalid base image name (line 2)
Dockerfile:2
--------------------
   1 |     ARG BASE_IMAGE
   2 | >>> FROM ${BASE_IMAGE}
   3 |     
   4 |     # Add fortran
I am using
Copy code
docker_image(
    name="chemistry",
    dependencies=[
        ":deps",
        "infra/base-images/images/base:base",
    ],
    extra_build_args=[
        "BASE_IMAGE=infra/base-images/images/base:base",
    ],
)
Copy code
pants dependencies //infra/base-images/images/chemistry:chemistry
infra/base-images/images/base:base
infra/base-images/images/chemistry/mamba_environment.yaml:deps
infra/base-images/images/chemistry/pip_requirements.txt:deps
infra/base-images/images/chemistry/pip_requirements.txt:reqs
infra/base-images/images/base:base
is building correctly
I found the issue.
Copy code
ARG BASE_IMAGE
this alone without a default value doesn't make it work, even if I always supply the value as build-args
very strange!
b
ah, i don't know if that's a pants issue or a docker issue. If you feel inclined and do more debugging (e.g. with 
--keep-sandboxes=always
to strip away the pants layer) and find it's a pants issue, please file a bug!
s
so it was a combination of issues. finally sorted for good. main culprit is that buildx con github actions uses "docker_containerd" instead of "docker" as an engine, so this was needed:
Copy code
- uses: docker/setup-buildx-action@v3
      with:
        install: true
        driver: docker
because https://github.com/docker/buildx/issues/301#issuecomment-755164475
Copy code
The docker-container driver you need to use for multi-platform builds doesn't support accessing local images, so you need to pull from a registry, whether local or remote.
discussion: https://github.com/pantsbuild/pants/issues/20189#issuecomment-1878162989 solution: https://github.com/pantsbuild/example-docker/pull/12/files
I would make a very clear change to the documentation to explain more in details how using reference docker images works
b
The docker docs are in https://github.com/pantsbuild/pants/tree/main/docs/docs/docker, if you have concrete ideas! Contributions desired! 🙂
5 Views
Open in Slack
PreviousNext
Powered by