Follow up question regarding my docker / pants compatibility issues, maybe someone has a fix for that. I need to use docker in docker for CI purpose. The official docker in docker is using an alpine base, and it doesn’t look like there is an official alternative. However, pants is not build for alpine I haven’t find much on github, pretty much just one comment is there anyone having a similar issue, and if so, any workaround?

you mean, you're trying to run pants inside a docker image to build another docker image?

Pretty much yeah. Gitlab recommend docker in docker for CI pipeline that need to build a docker image yeah

i think those images get trashed after finishing, right?

We use docker in docker regularly for our builds with debian/ubuntu base images and it seems to be fine. Although we’re not currently using the pants docker integration for building the ‘inner’ image

I'm not sure that's a recommendable way of how to use pants to begin with. you want to cache build artifacts and I don't know if generating new images to connect to the cache is a good practice.

I’m not sure I follow. Docker still has a long-running daemon on the host, what we’re doing here is tunneling the way in to the host daemon into the builder image

that's why i asked if gitlab trashes the CI image once the build job finished

Ah that makes sense

can multiple runners access the same cache at the same time without conflicting?

Good question, I’ve never tried 😛

i wouldn't believe so

to add on top, I think i remember that Gitlab has some form of distributed artifact cache. I feel very uncomfortable thinking about running pants like this. I would use it as a dedicated build server.

Not sure what you mean by distributed. In our case, we’re using s3 as a storage for artefact caching. Which is not “distributed” (as, there is a single source of truth), however, the runners are distributed yes, using a single cache source

@agreeable-oyster-28981 you have to ask yourself the question why and how you want to use Pants. Typically Pants runs in a long lived process while caching artifacts and invalidating what changed to reduce work. I doubt that using Gitlab's artifact cache integrates well into that and trashing the container after a successful run prevents you from using all the goodies Pants can offer.

I’m curious, what’s the reason for that?

We use pulumi to manage our infrastructure, with a custom pants plugin to run the program inside an isolated context orchestrated by pants (pulumi is essentially terraform but you can use python instead of a markup language). Pulumi has a docker.Image which when used will build the docker image and push it to its registry at build time. All we have to do is add the dockerfile as a loose

files()

context as a dependency to the custom pulumi target. We’ve been doing this before pants rolled out a

docker_image

target, and just haven’t felt a compelling reason to switch 🤷

Actually, having to fetch the toolchains every time you run a CI job sounds like a disadvantage. You really might want to rethink how you use your CI, or Pants

I think that we’re not looking for the same benefits from pants. the main benefit for us is to have a convenient way to setup a python mono-repo. I gave poetry a try but it was falling behind when trying to integrate with Google Cloud Function and building a docker image. So far, pants have been pretty good for all that. I’ll check what needs to be cached for pants to run properly, but I don’t see any reason why that would not work, every other build tool I’ve used (mainly yarn and npm) works properly with Gitlab caching. Maybe I should have been more accurate about caching. It’s not “artifact caching” per se. Artifact are stored when a job is complated. For the “caching” part, it’s not just the “artifacts” that are cached, but pretty much any files. So

~/.cache/pants

can be added properly regarding the toolchain, nothing prevent me to have it part of my CI image, I was actually considering testing that at some points.

i didn't say it won't work, you just lose a lot of benefits 😉 nevertheless, did the ubuntu image work?

gitlab/gitlab-runner:latest

didn’t really help. I just installed docker directly and will mount the socket as suggested by @faint-businessperson-86903 I was trying to follow the docker:dind pattern, using a

docker-entrypoint.sh

file, which works if I’m tyring to do direct call to docker, as

docker build . xxx

but doesn’t work when using pants through docker, as it bypass the entry point.