What context does pants uses when building docker ...
# general
What context does pants uses when building docker images? In this case I have a BUILD file inside the
with this target.
Copy code
This is a dockerfile where I am adding a
Copy code
FROM python:3.8.1-slim
COPY requirements.txt /usr/src/app
When I run
./pants package <path_to_docker_file>
, it can't find the file inside the
. This is the error stack
Copy code
./pants package src/pyfleet-vehicle-spec:docker_dev
15:37:13.87 [INFO] Completed: Building docker image docker_dev:latest.
15:37:13.87 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building docker image docker_dev:latest.' failed with exit code 1.

#1 [internal] load build definition from Dockerfile_dev
#1 sha256:70afa1e12870fb24144364708587cf5d1661f981d416ff91b9933fb09bf718bf
#1 transferring dockerfile: 370B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:8be3f6d6908bac7ae4d1c83b3c98641278b5369f05b904a5bd61c9aa771021ab
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for <http://docker.io/library/python:3.8.1-slim|docker.io/library/python:3.8.1-slim>
#3 sha256:897c30c5b0df12bcc7293f681243e639405b0ad6ce158f8ac7f4d6aa75160f3a
#3 DONE 0.7s

#4 [1/2] FROM <http://docker.io/library/python:3.8.1-slim@sha256:73f3903470a6e55202a6bb989c23b047487eb1728feba655410076da24106838|docker.io/library/python:3.8.1-slim@sha256:73f3903470a6e55202a6bb989c23b047487eb1728feba655410076da24106838>
#4 sha256:28d9be0d3eda75c94bdf3ffcb19aba18126f795be53f25d1fc3ed17716e5e83e

#5 [internal] load build context
#5 sha256:d8740fb29f3447aa31b7339dc3205fc961ed6fff4d68fe91e5f24b402262ebf3
#5 transferring context: 2B done
#5 DONE 0.0s

#6 [2/2] COPY requirements.txt /usr/src/app
#6 sha256:962d6b5ae5239ecc9e26d88d359e20065c2a3d56011710983dd0ee0face6bfd1
#6 ERROR: "/requirements.txt" not found: not found
 > [2/2] COPY requirements.txt /usr/src/app:
failed to compute cache key: "/requirements.txt" not found: not found
It will only include files in the context that you depend upon. Currently, dependency inference only picks up
targets from your
instructions, so you will have to add your
file to a
target, and depend on that from your
Improving feedback regarding the build context is high on my todo list.
This section of the docs are relevant regarding this question: https://www.pantsbuild.org/v2.9/docs/docker#adding-dependencies-to-your-docker_image-targets
If you’ve read that, and still don’t know what is what, I’d appreciate if you let me know, so we can improve upon it 🙂
🙏 1
yeah not sure if the documentation is the problem. I would add an example or two. So now as far as I see, I need to specify the path to dockerfile relative to the repo root instead of relative to the Dockerfile location. For a monorepo, I wanted to keep these paths relative to the different projects root. In this case relative to
. This way you can even manually build it directly inside
. I will have to change all the paths where we are adding local directory to docker image relative to pants. Not sure if something like
makes sense like we have
for python
👍 1
We also use
for testing.
Yeah.. this affects what you can include in your build context. There was such an option initially, perhaps worth introducing again. cc @happy-kitchen-89482
Hmm, since files can in theory be pulled in from all over the repo, we kinda have to support addressing them from the root. But do
targets work for this? https://www.pantsbuild.org/docs/reference-relocated_files
can't then we set
to repo root?
by the way does
work with say all the targets generated using some glob pattern?
Just started to look at using the Docker plugin… It would definitely be nice to not have to modify the file paths in the Dockerfile so that they work the same way when running
docker build
within the directory. @curved-television-6568 wondering if there’s been any progress on the file locations or an issue raised. I got
to work - I think this is ok as a workaround for now. Btw the docs and error reporting is really easy to follow. 🙂
❤️ 2
files can in theory be pulled in from all over the repo
In practice, we could inspect the paths actually present in the build context, and descend the tree as long as there’s no branching out. I.e. if all files reside in a tree rooted at
we could build from
rather than the repo root. However, we should only do this when explicitly asked for, to a) avoid confusion by being explicit about it and b) in case you pull in something else that is rooted in say
things will break, and being explicit we can provide more informative error messages.
@happy-kitchen-89482 does this sound like a fair trade-off/feature to you?
@nutritious-hair-72580 there’s not been an issue created for this yet. I think my suggestion above makes sense, however, and is relatively easy to implement, so if there’s no objections from Benjy, I’ll go ahead and add that later this week..
Sounds good to me, thanks!
Oh, and regarding
Improving feedback regarding the build context is high on my todo list.
There are some improvements out now in 2.9.0 for Docker build context issues in case of build failures.
❤️ 1
I.e. if all files reside in a tree rooted at src/project/a/… we could build from ../a rather than the repo root.
Vs using
As an alternative approach to using
Okay. I'd bias towards not adding another feature and associated option.
is already the generic way to approach this problem and it has the benefit of working with all of Pants, including Python, Go, JVM, Shell. It's also documented already The surface area of Pants is already enormous and continues growing, so we probably want to be careful before adding a new feature when there's already a workaround
Ok. I’ll submit the PR anyway, as I’ve already implemented it. We can take this discussion further there (I’m about to sign off, so will get back tomorrow..)
👋 1
@hundreds-father-404 I’ve added some arguments for this new option in the PR. I don’t think that a growing surface area should be a limiting factor for new features, all new features should be well founded, naturally, and if there’s usable work arounds thoroughly considered before being added. However in this case, I don’t find the work around feasible enough for long term use. (see the PR for an simple example with a shell script, that doesn’t work, for instance) Back to the surface area (as that is the one topic you brought up that I didn’t address already in the PR), wouldn’t it be better to find ways to manage that to make each part crystal clear. I.e. if jvm and go gets huge, it would be cool if that didn’t cloud other unrelated backends/features. Perhaps there’s need for better grouping/nesting of features/goals/option scopes/etc as more and more gets added?
👀 1
👍 1
I’ve added some arguments for this new option in the PR.
Cool, will take a look! To clarify, I'm not blocking and haven't looked closely enough to say no or anything. Only to bias towards reusing existing functionality like
when feasible to do so.
Perhaps there’s need for better grouping/nesting of features/goals/option scopes/etc as more and more gets added?
Perhaps, although generally I think it's valuable to have mechanisms that apply across backends. For example: • source roots are the same mechanism regardless of Java vs Python •
work with multiple languages • Codegen targets like
work with multiple languages Although, I think it could be both. We try to have mechanisms that generalize to multiple backends, and we also try to have good grouping/nesting of features when that's not possible