https://pantsbuild.org/ logo
#general
Title
# general
p

proud-appointment-36730

06/22/2022, 10:09 PM
Hey all, having some trouble getting
pants publish
to work for docker images being published to an AWS ECR repo. I think I got all the environment variables and
$PATH
components wired in correctly, but no dice. I following the recommendation about using
env -i
and actually got that to work, but it's not working through
pants publish
for some reason. I set
--level=trace
in my command, and it seems like I'm missing a ton of info about how
docker push
is getting executed (i.e no indication of the sandbox directory, command args, etc in the log output). Any ideas?
The output error message is
no basic auth credentials
, image builds fine. SO for some reason it seems like an auth issue but I'm stumped as to why my config file setup isn't wiring in the required environment variables
Relevant config section:
Copy code
[docker]

default_repository = "build-system-demo-pants"

env_vars = [
    # "DOCKER_CONFIG=build_support/docker/config",
    "AWS_ACCESS_KEY_ID=<REDACTED>",
    "AWS_SECRET_ACCESS_KEY=<REDACTED>",
    "AWS_ECR_CACHE_DIR=/Users/me/code/build-system-demo/ecr-cache",
    "DOCKER_CONFIG=/Users/me/code/build-system-demo/build_support/docker/config"
]

tools = [
    "docker-credential-ecr-login",
    "sh",
]
f

fast-nail-55400

06/22/2022, 10:44 PM
add
--no-process-cleanup
to preserve the execution sandbox
p

proud-appointment-36730

06/22/2022, 10:48 PM
@fast-nail-55400 I did that but it doesn't print any information about an execution sandbox for the push operation
I see this for the build operation:
Copy code
18:49:11.34 [INFO] Preserving local process execution dir /private/var/folders/1r/ndd87ylx3097l5pjbj1pgsch0000gp/T/process-executioniMMn2k for "Building docker image <http://137296740171.dkr.ecr.us-west-2.amazonaws.com/build-system-demo-pants:latest|137296740171.dkr.ecr.us-west-2.amazonaws.com/build-system-demo-pants:latest>"
And if I got there the
__run.sh
script just has the build commands. No similar output or script for the push operation
f

fast-nail-55400

06/22/2022, 11:06 PM
ah publish uses interactive process (via
InteractiveProcess
in Pants code) which means it runs differently than normal execution sandbox processes
basic question: I assume ecr-login is configured as a credential helper?
p

proud-appointment-36730

06/23/2022, 1:38 AM
Yeah, here are the contents of the config file:
Copy code
{
  "credHelpers": {
    "<account-id>.<http://dkr.ecr.us-west-2.amazonaws.com|dkr.ecr.us-west-2.amazonaws.com>": "ecr-login"
  }
}
h

happy-kitchen-89482

06/23/2022, 6:23 AM
cc @curved-television-6568 on this one, sorry Andreas! 🙂
🙏 1
c

curved-television-6568

06/23/2022, 8:13 AM
So, to trouble shoot this is a little involved, but doable. And it seems @proud-appointment-36730 managed to get it going with
env -i
So my question then would be, what env vars was used in that case? The exact same set of env vars as listed in the
pants.toml
file? What did you provide as
PATH
? I think a key component here may be a mistake to provide the path directly to a binary where there also are other binaries, so those are “leaked” into the sandboxed environment. To test this properly, you need a temporary
bin
folder, from which you can link all your binaries/tools and then point your
PATH
to that temp
bin
so all that is visible on the path are explicitly only those linked files and nothing else.
Also, this is relevant: https://github.com/pantsbuild/pants/issues/14596#issuecomment-1142430052 Seems like it would be worth while to implement..
p

proud-appointment-36730

06/23/2022, 1:37 PM
@curved-television-6568 I did use the same exact set of env vars, and I did the temporary symlinking of binaries into a temporary folder. Let me send you my command line invocation, one moment
👍 1
Script for testing outside of pants:
Copy code
#!/bin/bash

echo "<account-id>.<http://dkr.ecr.us-west-2.amazonaws.com|dkr.ecr.us-west-2.amazonaws.com>" | env -i \
  PATH=/tmp/path-isolated \
  AWS_ECR_CACHE_DIR=/Users/kyle/kairos/build-system-demo/ecr-cache \
  AWS_ACCESS_KEY_ID=<REDACTED> \
  AWS_SECRET_ACCESS_KEY=<REDACTED> \
  DOCKER_CONFIG=/Users/kyle/kairos/build-system-demo/build_support/docker/config \
  docker-credential-ecr-login get
Contents of
/tmp/path-isolated
:
Copy code
$ l /tmp/path-isolated
total 0
lrwxr-xr-x  1 kyle  wheel    42B Jun 22 17:59 docker-credential-ecr-login@ -> /usr/local/bin/docker-credential-ecr-login
lrwxr-xr-x  1 kyle  wheel     7B Jun 22 17:59 sh@ -> /bin/sh
Oh, I didn't provide a
PATH
environment variable in my
pants.toml
file though
c

curved-television-6568

06/23/2022, 1:44 PM
Oh, I didn’t provide a
PATH
environment variable in my
pants.toml
file though
That you should get from the
tools
section for you, so not the issue..
However, adding
PATH
to the env vars would blow a big whole but ought to work, as workaround for now, if nothing else.
I’ll see if I can learn anything about the above..
Huh, I’m stumped. Given the above works, I can’t see why it wouldn’t when executed from Pants. Will have to try this with a live connection to aws to get any further. Well investigated, Kyle 👍
p

proud-appointment-36730

06/23/2022, 1:55 PM
Thanks! Yeah I'm stumped as well. Is there any way to hook into the environment that the
docker push
command is running in and interactively debug? Or could I run some one-off shell commands in the environment the
docker push
command is running in to inspect it? I took a look at the pants code, what's the reasoning for using an
InteractiveProcess
to run the
docker push
? The fact that I can't use
--no-process-cleanup
to drop into the environment and figure out what's going on is making this a little tricky
c

curved-television-6568

06/23/2022, 2:56 PM
Yeah, I’ve missed the sandbox introspection aspect a few times as well. I can’t really recall the real reason for going with
InteractiveProcess
for this atm, perhaps @witty-crayon-22786 has an answer for that.
But what it does is basically just a stripped down env, and then some trickery to setup shim scripts for the tools, to have on PATH. That’s it. So the debugging setup you’ve done already is as close to it as I know.
Adding
PATH
to the
[docker].env_vars
ought to punch through any obstacles though… to get unblocked for now, if acceptable.
f

fast-nail-55400

06/23/2022, 3:05 PM
Is there any way to hook into the environment that the
docker push
command is running in and interactively debug?
It will be more convoluted, but you can by modifying Pants source for the rules to add what options you need. So maybe increase `docker`’s log level for the push to get more info. Checkout a copy of Pants in a sibling directory to your repo and add a
pants_from_sources
script from one of the pantsbuild/example-* repos on GitHub. Then modify https://github.com/pantsbuild/pants/blob/adc76b4dd0b85feb6fbfcb6ed735e292244fe3ff/src/python/pants/backend/docker/util_rules/docker_binary.py#L96 to add
--log-level=debug
option to the push command line. Then run
./pants_from_sources
in your repo instead of
./pants
.
(We should probably add a Pants option to allow setting arguments on the
docker push
command line used. But for now, the above procedure will at least get you more information.)
👍 1
c

curved-television-6568

06/23/2022, 3:08 PM
Actually, to capture/modify what options is used, the env etc, you could create a
docker
shim, put that on your path so Pants picks it up instead of the real docker binary. That way you can debug everything via that shim.
using
PANTS_DOCKER_EXECUTABLE_SEARCH_PATHS
.
f

fast-nail-55400

06/23/2022, 3:09 PM
good idea. no need for modifying Pants source then.
1
(given I work on Pants, too easy for me to just start editing Pants …)
😅 2
p

proud-appointment-36730

06/23/2022, 3:11 PM
Both ways honestly seem great. One thing I noticed is I don't think
pants publish
is actually using the environment variables in my config file. If it were, it would be writing to the log file in
/Users/kyle/kairos/build-system-demo/ecr-cache
. When using the
env -i
approach, I can see logs get streamed in if I
tail -f
the log file. When using pants, no logs arrive
c

curved-television-6568

06/23/2022, 3:12 PM
Oh..
p

proud-appointment-36730

06/23/2022, 3:13 PM
My config snippet is at the top of this thread, is there a formatting issue there or something?
👀 1
c

curved-television-6568

06/23/2022, 3:14 PM
looks good. I’ll run some tests…
env_vars
seems to be picked up, verified using the docker shims technique..
Copy code
$ PANTS_DOCKER_EXECUTABLE_SEARCH_PATHS='["/Users/x/src/tmp/shims"]' PANTS_DOCKER_ENV_VARS='["foo=bar", "LOGNAME"]' ./pants publish testprojects/src/python/docker:test-example
17:21:58.70 [INFO] Starting: Building docker image test-example:1.2.5
17:21:58.71 [INFO] Canceled: Building docker image test-example:1.2.5
17:21:59.10 [INFO] Starting: Building docker image test-example:1.2.5
17:21:59.12 [INFO] Completed: Building docker image test-example:1.2.5
17:21:59.13 [INFO] Built docker image: test-example:1.2.5
DOCKER SHIM!!
Env:
PWD=/private/tmp/.tmpBzbEdb
foo=bar
SHLVL=1
LOGNAME=x
_=/usr/bin/env
exec: docker push test-example:1.2.5

✓ test-example:1.2.5 published.
And in `…/shims/docker`:
Copy code
#!/bin/bash

echo "DOCKER SHIM!!"
echo "Env:"
/usr/bin/env
echo "exec: docker $@"
p

proud-appointment-36730

06/23/2022, 3:38 PM
Huh, your command isn't working for me. It says it can't find the executable (I changed the path to the correct location). I'm running pants in ZSH, could this be some weird quirk of my shell?
c

curved-television-6568

06/23/2022, 3:39 PM
Huh, no idea. I’m not familiar with zsh. You can put those config options in your pants.toml however… just me that took a shortcut to avoid polluting it…
p

proud-appointment-36730

06/23/2022, 3:45 PM
HUh working now. It didn't like the shim being in my repo I guess
👍 1
Interestingly, publishing images works if I remove
[docker].tools
entirely from my config
😲 2
c

curved-television-6568

06/23/2022, 4:17 PM
That was…. counter intuitive. Also, which version of Pants are you using?
p

proud-appointment-36730

06/23/2022, 4:18 PM
Copy code
$ ./pants --version
12:17:53.96 [INFO] Initializing scheduler...
12:17:54.28 [INFO] Scheduler initialized.
2.11.0
👍 1
Thanks so much everyone for all your help. To tie off this loose end, here is the config that ended up successfully publishing images to ECR for me:
Copy code
[docker]

default_repository = "build-system-demo-pants"

env_vars = [
    "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY",
    "DOCKER_CONFIG='%(buildroot)s/build_support/docker/config'",
    "AWS_ECR_CACHE_DIR='%(buildroot)s/build_support/docker/ecr-cache'",
]

tools = [
    "sh",
]
I gotcha that I think is very interesting to note is that doing the following does not work:
Copy code
[docker]

default_repository = "build-system-demo-pants"

env_vars = [
    "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY",
    "DOCKER_CONFIG='%(buildroot)s/build_support/docker/config'",
    "AWS_ECR_CACHE_DIR='%(buildroot)s/build_support/docker/ecr-cache'",
]

tools = [
    "sh",
    "docker-credential-ecr-login",
]
So for some reason shimming in the credential helper in the case of ECR breaks things, and docker can't find the credentials it need
Another interesting note. If you set the environment variables
PANTS_DOCKER_EXECUTABLE_SEARCH_PATHS="['/Users/kyle/shims/']"
, then the code will look for all the values in the
[docker].tools
part of the config in
/Users/kyle/shims
. Not sure if this is intended behavior or not, just wanted to point it out for anyone else who is trying to get ECR stood up and/or debug docker commands and stumbles on this thread
🙏 1
c

curved-television-6568

06/23/2022, 4:42 PM
Thanks Kyle! Regarding the search paths, that is as intended. (could add
"<PATH>"
at the end there, to have it do a wider search, only looking in the shims dir first)
h

hundreds-father-404

06/23/2022, 6:20 PM
Glad it's working! Thank you @curved-television-6568 for helping 💜 To clarify, are there changes we should make to Pants? Like bugs, or have better documentation?
💜 1
c

curved-television-6568

06/23/2022, 6:42 PM
I think we could at least investigate why it didn’t work with the docker credential helper listed in
[docker].tools
, but started working without.
h

hundreds-father-404

06/23/2022, 6:43 PM
Cool. What would be helpful there? File a ticket?
c

curved-television-6568

06/23/2022, 6:44 PM
Yeah, that could be worth while to have a ticket for, to help persist the history and future work.
p

proud-appointment-36730

06/23/2022, 7:14 PM
I think a little blurb in the docs about this weird edge case for ECR would be helpful too. I stuck it in
[docker].tools
because the docs (which have the google credential helper as an example) have it in there. Maybe explaining the edge case for ECR, or recommending to start with it empty and only add things in as needed, would be a helpful addition?
f

fast-nail-55400

06/23/2022, 7:39 PM
Would you want to submit a PR (as docs are in-repo now)?
docs/markdown/Docker/
p

proud-appointment-36730

06/23/2022, 7:39 PM
totally I'll throw that on my to-do list
My approach that worked locally doesn't work in CI. I've given up on the credential helpers and am employing a workaround that writes the credentials to the config file in CI. Not sure what the deal with ECR credential helper is 🤷
f

flat-summer-8204

06/28/2022, 2:46 PM
I am having the same issue
f

fast-nail-55400

06/28/2022, 2:56 PM
Was there anything in the ecr-login log directory? There should be a file called
ecr-login.log
under the
logs
directory in the ECR cache dir. (As suggested by the source code at https://github.com/awslabs/amazon-ecr-credential-helper/blob/main/ecr-login/config/log.go.)
p

proud-appointment-36730

06/28/2022, 2:58 PM
If you run it through pants, iirc it doesn't write to that file. I'm not sure why, other than some vague notions about pants stripping out environment variables that the credential help relies on for determining where to write logs to
f

fast-nail-55400

06/28/2022, 4:34 PM
I suggest explicitly setting
AWS_ECR_CACHE_DIR
in
[docker].env_vars
. Then the log should go in the
logs
subdirectory of that directory.
or the default should be
~/.ecr/logs
but would require
HOME
to be passed into the execution sandbox so the code can expand the path. https://github.com/awslabs/amazon-ecr-credential-helper/blob/e6f29200ae0450ba6584aee3041e2527e4ce1873/ecr-login/config/cache_dir.go#L22
p

proud-appointment-36730

07/01/2022, 6:14 PM
Yeah I thought I tried that with an explicit absolute path and didn't have much luck. I could have been missing something in my config though