https://pantsbuild.org/ logo
Join Slack
Powered by
What context does pants uses when building docker ...
# general
r
What context does pants uses when building docker images? In this case I have a BUILD file inside the 
src/pyfleet-vehicle-spec
with this target.
Copy code
docker_image(
    name="docker_dev",
    source="Dockerfile_dev"
)
This is a dockerfile where I am adding a 
requirements.txt
Copy code
FROM python:3.8.1-slim
COPY requirements.txt /usr/src/app
When I run 
./pants package <path_to_docker_file>
, it can't find the file inside the 
src/pyfleet-vehicle-spec
. This is the error stack
Copy code
./pants package src/pyfleet-vehicle-spec:docker_dev
15:37:13.87 [INFO] Completed: Building docker image docker_dev:latest.
15:37:13.87 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building docker image docker_dev:latest.' failed with exit code 1.
stdout:

stderr:
#1 [internal] load build definition from Dockerfile_dev
#1 sha256:70afa1e12870fb24144364708587cf5d1661f981d416ff91b9933fb09bf718bf
#1 transferring dockerfile: 370B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:8be3f6d6908bac7ae4d1c83b3c98641278b5369f05b904a5bd61c9aa771021ab
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for <http://docker.io/library/python:3.8.1-slim|docker.io/library/python:3.8.1-slim>
#3 sha256:897c30c5b0df12bcc7293f681243e639405b0ad6ce158f8ac7f4d6aa75160f3a
#3 DONE 0.7s

#4 [1/2] FROM <http://docker.io/library/python:3.8.1-slim@sha256:73f3903470a6e55202a6bb989c23b047487eb1728feba655410076da24106838|docker.io/library/python:3.8.1-slim@sha256:73f3903470a6e55202a6bb989c23b047487eb1728feba655410076da24106838>
#4 sha256:28d9be0d3eda75c94bdf3ffcb19aba18126f795be53f25d1fc3ed17716e5e83e
#4 CACHED

#5 [internal] load build context
#5 sha256:d8740fb29f3447aa31b7339dc3205fc961ed6fff4d68fe91e5f24b402262ebf3
#5 transferring context: 2B done
#5 DONE 0.0s

#6 [2/2] COPY requirements.txt /usr/src/app
#6 sha256:962d6b5ae5239ecc9e26d88d359e20065c2a3d56011710983dd0ee0face6bfd1
#6 ERROR: "/requirements.txt" not found: not found
------
 > [2/2] COPY requirements.txt /usr/src/app:
------
failed to compute cache key: "/requirements.txt" not found: not found
c
It will only include files in the context that you depend upon. Currently, dependency inference only picks up 
pex_binary
targets from your 
Dockerfile
COPY
instructions, so you will have to add your 
requirements.txt
file to a 
files
target, and depend on that from your 
docker_image
.
Improving feedback regarding the build context is high on my todo list.
This section of the docs are relevant regarding this question: https://www.pantsbuild.org/v2.9/docs/docker#adding-dependencies-to-your-docker_image-targets
If you’ve read that, and still don’t know what is what, I’d appreciate if you let me know, so we can improve upon it 🙂
🙏 1
r
yeah not sure if the documentation is the problem. I would add an example or two. So now as far as I see, I need to specify the path to dockerfile relative to the repo root instead of relative to the Dockerfile location. For a monorepo, I wanted to keep these paths relative to the different projects root. In this case relative to 
pyfleet-vehicle-spec
. This way you can even manually build it directly inside 
pyfleet-vehicle-spec
. I will have to change all the paths where we are adding local directory to docker image relative to pants. Not sure if something like 
docker_source_root
makes sense like we have 
source_root
for python
👍 1
We also use 
docker-compose
for testing.
c
Yeah.. this affects what you can include in your build context. There was such an option initially, perhaps worth introducing again. cc @happy-kitchen-89482
h
Hmm, since files can in theory be pulled in from all over the repo, we kinda have to support addressing them from the root. But do 
relocated_files
targets work for this? https://www.pantsbuild.org/docs/reference-relocated_files
r
can't then we set 
docker_source_root
to repo root?
by the way does 
relocated_files
work with say all the targets generated using some glob pattern?
n
Just started to look at using the Docker plugin… It would definitely be nice to not have to modify the file paths in the Dockerfile so that they work the same way when running 
docker build
within the directory. @curved-television-6568 wondering if there’s been any progress on the file locations or an issue raised. I got 
relocated_files
to work - I think this is ok as a workaround for now. Btw the docs and error reporting is really easy to follow. 🙂
❤️ 2
c
files can in theory be pulled in from all over the repo
In practice, we could inspect the paths actually present in the build context, and descend the tree as long as there’s no branching out. I.e. if all files reside in a tree rooted at 
src/project/a/…
we could build from 
../a
rather than the repo root. However, we should only do this when explicitly asked for, to a) avoid confusion by being explicit about it and b) in case you pull in something else that is rooted in say 
src/project/b/…
things will break, and being explicit we can provide more informative error messages.
@happy-kitchen-89482 does this sound like a fair trade-off/feature to you?
@nutritious-hair-72580 there’s not been an issue created for this yet. I think my suggestion above makes sense, however, and is relatively easy to implement, so if there’s no objections from Benjy, I’ll go ahead and add that later this week..
n
Sounds good to me, thanks!
c
Oh, and regarding
Improving feedback regarding the build context is high on my todo list.
There are some improvements out now in 2.9.0 for Docker build context issues in case of build failures.
❤️ 1
h
I.e. if all files reside in a tree rooted at src/project/a/… we could build from ../a rather than the repo root.
Vs using 
relocated_files
?
c
As an alternative approach to using 
relocated_files
h
Okay. I'd bias towards not adding another feature and associated option. 
relocated_files
is already the generic way to approach this problem and it has the benefit of working with all of Pants, including Python, Go, JVM, Shell. It's also documented already The surface area of Pants is already enormous and continues growing, so we probably want to be careful before adding a new feature when there's already a workaround
c
Ok. I’ll submit the PR anyway, as I’ve already implemented it. We can take this discussion further there (I’m about to sign off, so will get back tomorrow..)
👋 1
@hundreds-father-404 I’ve added some arguments for this new option in the PR. I don’t think that a growing surface area should be a limiting factor for new features, all new features should be well founded, naturally, and if there’s usable work arounds thoroughly considered before being added. However in this case, I don’t find the work around feasible enough for long term use. (see the PR for an simple example with a shell script, that doesn’t work, for instance) Back to the surface area (as that is the one topic you brought up that I didn’t address already in the PR), wouldn’t it be better to find ways to manage that to make each part crystal clear. I.e. if jvm and go gets huge, it would be cool if that didn’t cloud other unrelated backends/features. Perhaps there’s need for better grouping/nesting of features/goals/option scopes/etc as more and more gets added?
👀 1
👍 1
h
I’ve added some arguments for this new option in the PR.
Cool, will take a look! To clarify, I'm not blocking and haven't looked closely enough to say no or anything. Only to bias towards reusing existing functionality like 
relocated_files
when feasible to do so.
Perhaps there’s need for better grouping/nesting of features/goals/option scopes/etc as more and more gets added?
Perhaps, although generally I think it's valuable to have mechanisms that apply across backends. For example: • source roots are the same mechanism regardless of Java vs Python • 
resources
work with multiple languages • Codegen targets like 
protobuf_source
work with multiple languages Although, I think it could be both. We try to have mechanisms that generalize to multiple backends, and we also try to have good grouping/nesting of features when that's not possible
➕ 1
13 Views
Open in Slack
PreviousNext
Powered by