https://pantsbuild.org/ logo
#general
Title
# general
a

ancient-rose-27306

08/23/2022, 2:10 PM
I am trying to build a Docker image for one of my sources and getting following error:
Copy code
(pants) [root@9a5752857756 pants-training]# ./pants package src/server:docker
09:52:28.41 [INFO] Completed: Building dockerfile_parser.pex from dockerfile-parser_default.lock
09:52:32.72 [INFO] Completed: Building docker image docker:latest
09:52:32.73 [WARN] Docker build failed for `docker_image` src/server:docker. The src/server/Dockerfile have `COPY` instructions where the source files may not have been found in the Docker build context.

The following files were not found in the Docker build context:

  * ./3rdparty/requirements.txt
  * ./src/server/


09:52:32.73 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building docker image docker:latest' failed with exit code 1.
stdout:

stderr:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?



Use `--no-process-cleanup` to preserve process chroots for inspection.

(pants) [root@9a5752857756 pants-training]#
My source can be found here. It seems like I need to add some dependencies to the
docker_image()
target which I already tried but it did not work. Could anyone please help me with that?
s

sparse-lifeguard-95737

08/23/2022, 2:12 PM
yes, your
docker_image
target would need to define explicit dependencies on the requirements file and the
python_sources
target to build as you’ve defined it
instead of copying the requirements.txt and raw sources into your image, have you considered copying in the
pex_binary
you’re defining? there is some dependency inference support between
docker_image
and
pex_binary
targets
2
and if
greeter_server
isn’t using all the dependencies in your requirements file, building the pex will strip away anything that isn’t needed
a

ancient-rose-27306

08/23/2022, 2:14 PM
I already have the dependency on
python_sources()
target:
dependencies=[":server"]
line #20 in
src/server/BUILD
h

hundreds-father-404

08/23/2022, 2:14 PM
Thanks Dan. +1 to considering using
pex_binary
with Docker. This blog explains the integration: https://blog.pantsbuild.org/pants-pex-and-docker/
👍 1
a

ancient-rose-27306

08/23/2022, 2:15 PM
My current focus is to be able to build a Docker image successfully. The part where I need to consider using pex/python source comes later for me.
👍 1
As I mentioned, I already have the dependency on
python_sources()
target and I tried adding the reqs dependency but it did not work.
h

hundreds-father-404

08/23/2022, 2:16 PM
Re what you currently have, there is a major gotcha that we only add
dependencies
for `file`/`files` targets and then anything that can be `package`d, like a
pex_binary
target. So, unfortunately, the
python_sources
dependency gets ignored -- big gotcha If you want to stick with the current approach of copying those two files, you'll need to create
file
/
files
targets for the relevant code. For example:
Copy code
files(name="as_files", sources=["*.py"])

docker_image(name="docker", dependencies=[":as_files", "3rdparty:requirements_file"])
@bitter-ability-32190 has been doing a lot of thinking on how to improve the
dependencies
field. This is one example of many where it's confusing. A smaller precursor ticket for this gotcha is https://github.com/pantsbuild/pants/issues/16522 to at least document it
a

ancient-rose-27306

08/23/2022, 2:21 PM
Thanks @hundreds-father-404, I read about the Docker context and its expectation of a
file
/
files
target in the Pants Docker docs but wasn’t sure where do I define those targets? Moreover, I then read about these two target types and it said that any files “that lives outside of code packages” and that confused me further. For my first question, do I define a file target each in
3rdparty/BUILD
and
src/server/BUILD
for
requirements.txt
file and for
.py
file, respectively?
h

hundreds-father-404

08/23/2022, 2:21 PM
For my first question, do I define a file target each in 3rdparty/BUILD and src/server/BUILD for requirements.txt file and for .py file, respectively?
Yeah, exactly. And keep all your existing targets you have
1
a

ancient-rose-27306

08/23/2022, 2:59 PM
Ok, so I was able to define the
file
targets successfully but then ran into an error (attached). This seems more of a python package installation error and not Pants error but just wanted to share and request if you are able to pull the latest source and try
./pants package src/server:docker
?
h

hundreds-father-404

08/23/2022, 3:13 PM
yeah I think that is a Python error. It's because you put
grpc
in your
requirements.txt
, which we don't want. That's a bad package and
grpcio
needs to be used instead. So, delete
grpc
Then, to work around Pants's issue complaining it can't find
grpc
, you will want to set in
3rdparty/BUILD
in the
python_requirements
target the field
module_mapping={"grpcio": ["grpc"]}
-- you can delete that once we do the 2.12 release today
a

ancient-rose-27306

08/23/2022, 3:16 PM
Let me try.
So Docker build is successful too. I have two follow up questions regrading the workaround: 1. We introduced
grpc
in
requirements.txt
because my
python_distribution()
build was failing. I believe the
module_mapping
workaround would have worked in that case too. Please confirm. 2. How do I make sure that I have the latest code once 2.12 releases today? Would Pants automatically pull the latest updates because my
pants.toml
points to
2.12.0
.
h

hundreds-father-404

08/23/2022, 3:37 PM
Yeah, the
module_mapping
solution is the correct one. You are teaching Pants that the requirement named
grpcio
provides the module
grpc
How do I make sure that I have the latest code once 2.12 releases today? Would Pants automatically pull the latest updates because my pants.toml points to 2.12.0.
No, you must manually change
pants_version
in
pants.toml
to e.g.
2.12.1rc1
. We announce releases in #announce and the google group https://www.pantsbuild.org/docs/getting-help#mailing-list
👍 1
a

ancient-rose-27306

08/23/2022, 6:50 PM
The Docker image doesn’t have the required stubs in it. Following is the
docker-image()
target definition:
docker_image(
name="docker",
dependencies=[":server_source_code", "3rdparty:python_requirements"]
)
The dependencies are those files that need to be copied explicitly in the image. The
server_source_code
target has import statements to import protobufs. Why aren’t the stubs generated?
Also, how do I
publish
to Docker Hub? As in where do I provide the URL and credentials?
I think I found the answer to my second question: https://www.pantsbuild.org/docs/tagging-docker-images#configuring-registries Still trying to figure out the first one though which is to be able to copy the generated stubs in Dockerfile. Do I mention a dependency on
protos
or
src/server:server
here? Or do I just include a copy statement in Dockerfile and it will automatically generate the protos and copy it to Docker?
h

hundreds-father-404

08/23/2022, 9:21 PM
Pardon, I was in meetings. Ah, so I actually don't think there's currently a way to get the generated Protobuf files into the Docker image. The only hack I could think of is using
export-codegen
and manually saving the files to disk, but that goes against the point of Pants's codegen (ensuring you always have up-to-date files and don't need to check them in) +1 that the Docker backend really works substantially better when combining it with
pex_binary
. The Pex will already include the generated code for you, for example
a

ancient-rose-27306

08/23/2022, 9:40 PM
Ok, and packing a library to an image?
h

hundreds-father-404

08/23/2022, 9:45 PM
What do you intend to do with the image? PEXes are fairly flexible: you can set an entry point so that it's runnable, or leave off an entry point where running it opens a REPL with the relevant code in PYTHONPATH, or even call
unzip
on the file. PEXes are basically runnable zip files of Python code
a

ancient-rose-27306

08/23/2022, 9:49 PM
Yeah I understand that. Ok, let me first figure out if I need to pack a lib or a Pex to the Docker and get back to you. In the meanwhile, I am having probs in pushing image to the registry. I am working thru that.
👍 1
Is “latest” the default tag if I don’t provide any
image_tag
to
docker_image()
target?
I am trying
./pants --no-process-cleanup publish src/server:docker
and here is a brief snippet:
Copy code
17:55:57.47 [INFO] Preserving local process execution dir /tempdir/process-execution49vWpa for "Building docker image <https://url.com/repository-in-registry/docker:latest>"
17:55:59.39 [INFO] Completed: Building docker image <https://url.com/repository-in-registry/docker:latest>
17:55:59.39 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building docker image <https://url.com/repository-in-registry/docker:latest>' failed with exit code 125.
stdout:

stderr:
invalid argument "<https://url.com/repository-in-registry/docker:latest>" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
I have masked a lot of things like the name of the temp directory and the registry URL but have tried to keep the structure as same as possible. Looking at the logs, I want to ask if the image building is complete and whether it failed in the push to registry step? Following is the
docker_image()
target definition:
docker_image(
name="docker",
repository="repository-in-registry/docker",
dependencies=[":server_source_code", "3rdparty:python_requirements"],
registries=["@myregistry"]
pants.toml
is as follows:
[docker.registries.myregistry]
address = "<https://url.com>"
Good morning @hundreds-father-404, would you please be able to advise me with this today?
So I was able to run
./pants publish src/server:docker
successfully (as in it built the image). I figured I should not be using
https://
in the docker registry section in
pants.toml
. Is that a known thing? I’d like to contribute to the code or the docs to make it more apparent. Could you please guide me on that part? Also, the
publish
goal wasn’t “fully” successful as it failed to push it to the registry with the error:
unauthorized: unauthorized to access repository:
I am unable to find out anything in the docs that would point towards providing credentials to be able to push to a registry.
For now I have been able to fully execute the
publish
goal by doing
docker login -u <username> <registry-name>
but would like to know if there’s any config that can be done in
pants.toml
. I have found the documentation that talks about authentication in Docker but it doesn’t involve
pants.toml
.