Hello there, I'm having trouble executing

pants publish

in Google Cloud Build. I'm getting the following error:

unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials

However, this does not seem to be the case, since if I only use Pants to build the image (

pants package

), I can then push them with

docker push

, so the credentials seem to be in place. The

~/.docker/config.json

also looks correct. What am I missing here?

Have you checked the steps here? 🙂 https://www.pantsbuild.org/docs/docker#docker-authentication

I have, I'm doing it all. Not sure what else to do to make Pants pick the authentication.

👍 And did you get the troubleshooting command there to work as well, and matching what you have in your pants.toml?

What do you mean by this?

image.png

Oh right, but it doesn't seem that I'm missing tools. I had this problem too, and the error message said a tool was missing so I added it. Now the problem is with Pants not accessing the authentication credentials

Pants doesn't access the authentication credentials in any way to my knowledge. All it does is put together a sandbox with the right tools and environment for Docker to figure it out. That's why the above script is helpful, as it'd replicate what happens in the sandbox. Assuming for a moment that the configuration is indeed correct; it would indicate that the builder does not have the right permissions. Just to clarify that, when you say

package

+

push

works, is that using the same scripted flow? Does

pants publish

work locally?

The creds are in

~/.docker/config.json

?

not really. the suggestions here are all good.. still a bit of dirty work to get through these kinds of issues..

Thanks all for trying to help! •

package

+

push

work and

publish

doesn't, I run both in a Google Cloud Build pipeline • The creds are in

~/.docker/config.json

. I make sure this is the case by running

gcloud --auth configure-docker <http://gcr.io|gcr.io>

. This adds `gcr.io: glcoud`to

credHelpers

in the file.

If you can make a directory like

/tmp/test-creds

with the tools and eventually some external env variables (esp GOOGLE_APPLICATION_CREDENTIALS might be interesting) then that is the minimum set to make pants+docker work. The check is just that you get a credentials. For example; if I just put

docker-credential-gcr

in

/tmp/foobar

I get a first step:

echo europe-north1-docker.pkg.dev | env -i PATH=/tmp/foobar/ docker-credential-gcr
/tmp/foobar//docker-credential-gcr: 59: readlink: not found
/tmp/foobar//docker-credential-gcr: 73: dirname: not found
/tmp/foobar//docker-credential-gcr: 59: readlink: not found
/bin/gcloud: 59: readlink: not found
/bin/gcloud: 73: dirname: not found
/bin/gcloud: 59: readlink: not found
/bin/gcloud: 182: exec: python: not found

So starting from that I can see I need dirname, readlink, and python. Adding those I get a credential. Knowing that, and having those in the pants configuration, the likely candidate is the ~/.docker/config.json file not being included.

Thanks! But what exactly do you mean by the

~/.docker/config.json

file not being included? I mean, it is present in the Cloud Build container when the pipeline runs, I can

cat

it to see its contents.

Yes; but you need to tell Pants where it is too:

env_vars = ["DOCKER_CONFIG=%(homedir)s/.docker"]

.

Did that 🙂

Gotcha. If you run with

-ldebug --keep-sandboxes=on_failure

, you should see what commands Pants run and where it does so. Might be worthwhile to that there isn't some weird path or other funky thing happening there with variable expansions etc. It should (iirc) generate a run script in the sandbox too.

I am right thinking that

keep-sandboxes

allows to analyze a failed Pants execution after it failed? If it does, the CloudBuild container is removed anyway, so I can't access these sandboxes I guess

Yes, indeed. Pants generates a temporary directory (normally under

/tmp/pants-sandbox-RANDOM/

) containing all files and tools used in a command run.

-ldebug

should still be helpful, at least. It definitely sounds like an environment/config mismatch, but I'm a bit at a loss for what it would be. Maybe doing an

export

and just forwarding anything that seems even remotely useful could work as a debugging step.

I used the debugging flag to learn that pants is running docker push with

DOCKER_CONFIG="/root/.docker"

. (I have set

"DOCKER_CONFIG=%(homedir)s/.docker"

in pants.toml). But I'm not sure whether /root/ is where my docker config is. During CloudBuild execution, the HOME env var is set to

/builder/home

and I think that my valid docker configuration sits in

/builder/home/.docker

.

It's the expansion of

~

according to the docs. You could try

%(env.HOME)s

which should be $HOME.

Thank you! That solved it 🙂

would be great to capture your experiences here in a more future friendly consumable form.. 🙂

So $home isn't ~? That sounds... Interesting. Don't think I've had that happen before.

Sure! Here it is: • in Google Cloud Build pipelines, the $HOME directory is set to

/builder/home

and this is where docker configuration is stored (e.g. config file that is created when one runs

gcloud --auth configure-docker <http://gcr.io|gcr.io>

•

"~"

on the other hand seems to resolve to

/root/

. Consequently, setting

"DOCKER_CONFIG=%(homedir)s/.docker"

in pants.toml made Pants look for the config in

/root/.docker

• The solution: use

%(env.HOME)s

in pants.toml

I think this is relevant? https://github.com/pantsbuild/pants/pull/18465

If anyone is using Google Artifact Registry I found I had to use

docker-credential-gcloud

instead of

docker-credential-gcr

. I think Google is trying to move folks from container registry to artifact registry.