https://pantsbuild.org/ logo
Join Slack
Powered by
HI! I am using pants to build docker images (my te...
# general
s
HI! I am using pants to build docker images (my team loves pants ❤️) and now starting to optimise some processes around it. I am finding it takes a long time to run even if nothing in the image changes 🧵
For instance, with a clean git
Copy code
time pants package projects/mofgen:mofgen-default-image
real    0m28.940s
user    0m4.785s
sys     0m0.715s
I would expect it to understand that the context hasn't changed, and that the image is there, and not go any further?
Even more interestingly, if I remove the 
cache_to
and 
cache_from
directives, it becomes faster!
Copy code
real    0m12.534s
user    0m4.664s
sys     0m0.691s
is there anything I could do to make pants to be faster?
h
Would need more details, but I believe we always rerun docker builds because they side-effect (they write images into your local docker image cache)
For example, if you were to nuke that local image cache, Pants would have no way of knowing
It could make sense to have a flag that says 
--trust-me-i-have-not-messed-with-my-local-cache
or whatever 🙂
e
@happy-kitchen-89482 I've been under the impression that pants just throws the entire build at docker and most caching/speedup is just due to docker layer caching. Is that correct?
h
Yes, correct. The “cache” I am referring to above is docker’s layer cache
It is tricky to get a system like pants to work correctly in the presence of external side effects, such as the contents of that cache
s
Perhaps I will give more context 🙂 Maybe there is another way. I am building docker images, and then I need to register them with a remove application (flyte), for that, I need the image reference. I am trying to get the 
docker-info.json
using the "wrap_as_resource" and it "works", but it's incredibly slow, because every time it needs to re-run the docker build step
is there another way I could get the reference for an image I have already built?
h
I vaguely recall that we write some info out to a json file, would need to look in the source to see if that’s the case or that is an incepted memory
e
This is correct. 
dist/dotted.path.to.docker_image.target/target_name.docker-info.json
s
I think it would be nice if pants could use the pants hash to check if the docker image for that context has already been produced, and skip the build?
h
I think it could - @curved-television-6568, @careful-address-89803 do you see a reason not to?
BTW this is what forces 
docker
commands to run every time
although that is not quite true, since that prevents the process result from being cached across pants runs, but the rule graph will still short-circuit as long as pantsd is alive, I don’t see any uncacheable rules in the docker backend.
Needs a closer look
But I think as it stands if you, say, 
docker image rm
and want Pants to repopulate the docker image cache you need to restart pantsd. Perhaps someone with more experience on the docker backend can weigh in?
But that said, I don’t see why we couldn’t check the image cache, as long as we have the expected sha of the image, before actually building
❤️ 1
c
Pants would need to include the hash as image build info and check against that on existing images. That would work.
h
Doesn’t it already, in docker-info.json?
c
I don't think it does. docker-info.json is more a post-build artifact, no? I would expect a check around here https://github.com/pantsbuild/pants/blob/bf355d6408bc508313d10d1ca6064e9271c2cbe9/src/python/pants/backend/docker/goals/package_image.py#L443 that would need to conditionally run that build image process.
the build context does have a hash value that could be passed on to the docker build so it's baked into the image so we can check against that on subsequent builds: https://github.com/pantsbuild/pants/blob/bf355d6408bc508313d10d1ca6064e9271c2cbe9/src/python/pants/backend/docker/util_rules/docker_build_context.py#L133
h
I mean the hash is in the post-build artifact, which Pants can cache and then check for
s
Do you expect to work on this improvement? Is there a github issue for it, or would you like me to create one?
@happy-kitchen-89482 @curved-television-6568 sorry just trying to learn how to be a good community citizen. do you want me to open an issue with this?
h
Hey @stale-waitress-56895, feel free to open an issue for further discussion. I don’t expect anyone to work on this unless they are motivated by their own needs (e.g, either you do the work or you pay someone to do it), but a way to start generating interest is a GH issue. If you’re interested in implementing we can guide you through it.
✅ 1
b
Somehow related to this thread: https://github.com/pantsbuild/pants/issues/22390
8 Views
Open in Slack
PreviousNext
Powered by