https://pantsbuild.org/ logo
#general
Title
# general
c

clever-father-91273

03/10/2023, 12:53 PM
Hello there, I'm having trouble executing
pants publish
in Google Cloud Build. I'm getting the following error:
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials
However, this does not seem to be the case, since if I only use Pants to build the image (
pants package
), I can then push them with
docker push
, so the credentials seem to be in place. The
~/.docker/config.json
also looks correct. What am I missing here?
g

gorgeous-winter-99296

03/10/2023, 1:16 PM
Have you checked the steps here? šŸ™‚ https://www.pantsbuild.org/docs/docker#docker-authentication
It does sound like it might not see all the tools or docker configurations.
c

clever-father-91273

03/10/2023, 1:17 PM
I have, I'm doing it all. Not sure what else to do to make Pants pick the authentication.
g

gorgeous-winter-99296

03/10/2023, 1:31 PM
šŸ‘ And did you get the troubleshooting command there to work as well, and matching what you have in your pants.toml?
c

clever-father-91273

03/10/2023, 1:33 PM
What do you mean by this?
g

gorgeous-winter-99296

03/10/2023, 1:33 PM
image.png
c

clever-father-91273

03/10/2023, 1:35 PM
Oh right, but it doesn't seem that I'm missing tools. I had this problem too, and the error message said a tool was missing so I added it. Now the problem is with Pants not accessing the authentication credentials
g

gorgeous-winter-99296

03/10/2023, 2:52 PM
Pants doesn't access the authentication credentials in any way to my knowledge. All it does is put together a sandbox with the right tools and environment for Docker to figure it out. That's why the above script is helpful, as it'd replicate what happens in the sandbox. Assuming for a moment that the configuration is indeed correct; it would indicate that the builder does not have the right permissions. Just to clarify that, when you say
package
+
push
works, is that using the same scripted flow? Does
pants publish
work locally?
h

happy-kitchen-89482

03/10/2023, 3:03 PM
The creds are in
~/.docker/config.json
?
cc @curved-television-6568 in case this reminds him of anything
c

curved-television-6568

03/10/2023, 3:47 PM
not really. the suggestions here are all good.. still a bit of dirty work to get through these kinds of issues..
c

clever-father-91273

03/10/2023, 3:51 PM
Thanks all for trying to help! ā€¢
package
+
push
work and
publish
doesn't, I run both in a Google Cloud Build pipeline ā€¢ The creds are in
~/.docker/config.json
. I make sure this is the case by running
gcloud --auth configure-docker <http://gcr.io|gcr.io>
. This adds `gcr.io: glcoud`to
credHelpers
in the file.
@gorgeous-winter-99296 How would you suggest to use the troubleshooting command you mentioned? I mean, it can be used to check what tools are present, but what should I check?
Re: 1st bullet above: locally
publish
works; I'm only having trouble on Cloud Build.
g

gorgeous-winter-99296

03/10/2023, 4:05 PM
If you can make a directory like
/tmp/test-creds
with the tools and eventually some external env variables (esp GOOGLE_APPLICATION_CREDENTIALS might be interesting) then that is the minimum set to make pants+docker work. The check is just that you get a credentials. For example; if I just put
docker-credential-gcr
in
/tmp/foobar
I get a first step:
Copy code
echo europe-north1-docker.pkg.dev | env -i PATH=/tmp/foobar/ docker-credential-gcr
/tmp/foobar//docker-credential-gcr: 59: readlink: not found
/tmp/foobar//docker-credential-gcr: 73: dirname: not found
/tmp/foobar//docker-credential-gcr: 59: readlink: not found
/bin/gcloud: 59: readlink: not found
/bin/gcloud: 73: dirname: not found
/bin/gcloud: 59: readlink: not found
/bin/gcloud: 182: exec: python: not found
So starting from that I can see I need dirname, readlink, and python. Adding those I get a credential. Knowing that, and having those in the pants configuration, the likely candidate is the ~/.docker/config.json file not being included.
c

clever-father-91273

03/10/2023, 4:08 PM
Thanks! But what exactly do you mean by the
~/.docker/config.json
file not being included? I mean, it is present in the Cloud Build container when the pipeline runs, I can
cat
it to see its contents.
g

gorgeous-winter-99296

03/10/2023, 4:08 PM
Yes; but you need to tell Pants where it is too:
env_vars = ["DOCKER_CONFIG=%(homedir)s/.docker"]
.
c

clever-father-91273

03/10/2023, 4:09 PM
Did that šŸ™‚
g

gorgeous-winter-99296

03/10/2023, 4:13 PM
Gotcha. If you run with
-ldebug --keep-sandboxes=on_failure
, you should see what commands Pants run and where it does so. Might be worthwhile to that there isn't some weird path or other funky thing happening there with variable expansions etc. It should (iirc) generate a run script in the sandbox too.
c

clever-father-91273

03/10/2023, 4:15 PM
I am right thinking that
keep-sandboxes
allows to analyze a failed Pants execution after it failed? If it does, the CloudBuild container is removed anyway, so I can't access these sandboxes I guess
g

gorgeous-winter-99296

03/10/2023, 4:26 PM
Yes, indeed. Pants generates a temporary directory (normally under
/tmp/pants-sandbox-RANDOM/
) containing all files and tools used in a command run.
-ldebug
should still be helpful, at least. It definitely sounds like an environment/config mismatch, but I'm a bit at a loss for what it would be. Maybe doing an
export
and just forwarding anything that seems even remotely useful could work as a debugging step.
c

clever-father-91273

03/10/2023, 5:19 PM
I used the debugging flag to learn that pants is running docker push with
DOCKER_CONFIG="/root/.docker"
. (I have set
"DOCKER_CONFIG=%(homedir)s/.docker"
in pants.toml). But I'm not sure whether /root/ is where my docker config is. During CloudBuild execution, the HOME env var is set to
/builder/home
and I think that my valid docker configuration sits in
/builder/home/.docker
.
What is Pants using as a replacement for
%(homedir)s
, is it not
$HOME
?
g

gorgeous-winter-99296

03/10/2023, 5:22 PM
It's the expansion of
~
according to the docs. You could try
%(env.HOME)s
which should be $HOME.
šŸ™Œ 1
c

clever-father-91273

03/10/2023, 5:38 PM
Thank you! That solved it šŸ™‚
šŸ‘ 1
c

curved-television-6568

03/10/2023, 5:41 PM
would be great to capture your experiences here in a more future friendly consumable form.. šŸ™‚
g

gorgeous-winter-99296

03/10/2023, 5:43 PM
So $home isn't ~? That sounds... Interesting. Don't think I've had that happen before.
c

clever-father-91273

03/10/2023, 5:45 PM
Sure! Here it is: ā€¢ in Google Cloud Build pipelines, the $HOME directory is set to
/builder/home
and this is where docker configuration is stored (e.g. config file that is created when one runs
gcloud --auth configure-docker <http://gcr.io|gcr.io>
ā€¢
"~"
on the other hand seems to resolve to
/root/
. Consequently, setting
"DOCKER_CONFIG=%(homedir)s/.docker"
in pants.toml made Pants look for the config in
/root/.docker
ā€¢ The solution: use
%(env.HOME)s
in pants.toml
šŸ™ 2
h

happy-kitchen-89482

03/10/2023, 7:48 PM
s

silly-queen-7197

05/26/2023, 5:10 PM
If anyone is using Google Artifact Registry I found I had to use
docker-credential-gcloud
instead of
docker-credential-gcr
. I think Google is trying to move folks from container registry to artifact registry.
17 Views