https://pantsbuild.org/ logo
Join Slack
Powered by
Has anyone got a clue why `pants publish docker/py...
# general
v
Has anyone got a clue why 
pants publish docker/python:base
would work fine publishing to google cloud but then if I delete the local version of the image (or go to a different machine with the same user credentials) then 
pants run server:docker
(which triggers pulling 
docker/python:base
from the repository) would fail with a permission denied error 
Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource
yet docker pull works. So somehow the authentication is working for pushing but then it fails for pulling in the pants sandbox.
w
I've never seen this problem before (using Azure, anyways), but how difficult would a repo of this problem be somewhere? Kinda weird, with the auth
v
Not so easy because you need a google cloud artifact repository etc
w
is it limited to google? Or have you seen this sort of thing happen on other services?
v
i haven't tried any others
w
👍 Its been a while, but I'd tried something similar on a few services, and some time ago it seemed to work, so I wonder if this is a situation with Google or a regression. What does 
-ldebug
have to say? any errors
v
yeah, it's an issue that went unnoticed for a while since if I publish first then the image is available and 
run
would just pull a local copy and everything works. I only noticed when someone else tried to pull first so there was no local copy. I'm trying to scan through ldebug, as far as i can tell, it's pulling in the same environment variables for pushing and run so still a mystery
Failing:
Copy code
spawned local process as Some(44041) for Process { argv: ["/usr/local/bin/docker", "buildx", "build", "--platform=linux/arm64,linux/amd64", "--output=type=docker", "--pull=False", "--tag", "us-docker.pkg.dev/jazmo-prod/docker/python:latest", "--build-arg", "DOCKER_ROOT_DIR=/jazmo", "--build-arg", "PORT=8080", "--file", "docker/python/Dockerfile", "."], env: {"CLOUDSDK_CONFIG": "/Users/mfairley/jazmoai/jazmo/.local/etc/gcloud", "DOCKER_CONFIG": "/Users/mfairley/jazmoai/jazmo/.local/etc/docker", "PATH": "/private/var/folders/lf/tr4gz0k514lf165m8gcg73yw0000gn/T/pants-sandbox-GnNOpy/_binary_shims_1c7d76e94cd5e5e5cf79a69210fe46a9d86d49e71077876afc1a42c6c025ebc7", "__UPSTREAM_IMAGE_IDS": ""}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<5928cdd3e3190dd5b13e041b275fcd6c070aac35b959a73f34b5029d942d1e52>, size_bytes: 234 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<30f996cdfdd32de948a5747fb779b51554b8e10eba86ac75c0474bd016a666bd>, size_bytes: 80 }, tree: "Some(..)" }, immutable_inputs: {RelativePath("_binary_shims_1c7d76e94cd5e5e5cf79a69210fe46a9d86d49e71077876afc1a42c6c025ebc7"): DirectoryDigest { digest: Digest { hash: Fingerprint<1c7d76e94cd5e5e5cf79a69210fe46a9d86d49e71077876afc1a42c6c025ebc7>, size_bytes: 555 }, tree: "Some(..)" }}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, concurrency: None, description: "Building docker image us-docker.pkg.dev/jazmo-prod/docker/python:latest", level: Info, append_only_caches: {}, jdk_home: None, cache_scope: PerSession, execution_environment: ProcessExecutionEnvironment { name: None, platform: Macos_arm64, strategy: Local, local_keep_sandboxes: Always }, remote_cache_speculation_delay: 0ns, attempt: 0 }
Working:
Copy code
Execute InteractiveProcess(process=Process(argv=('/usr/local/bin/docker', 'push', 'us-docker.pkg.dev/jazmo-prod/docker/python:latest'), description='Pushing docker image us-docker.pkg.dev/jazmo-prod/docker/python:latest', level=<LogLevel.INFO: 'info'>, input_digest=Digest('e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 0), immutable_input_digests=FrozenDict({'_binary_shims_1c7d76e94cd5e5e5cf79a69210fe46a9d86d49e71077876afc1a42c6c025ebc7': Digest('1c7d76e94cd5e5e5cf79a69210fe46a9d86d49e71077876afc1a42c6c025ebc7', 555)}), use_nailgun=(), working_directory=None, env=FrozenDict({'PATH': '{chroot}/_binary_shims_1c7d76e94cd5e5e5cf79a69210fe46a9d86d49e71077876afc1a42c6c025ebc7', 'CLOUDSDK_CONFIG': '/Users/mfairley/jazmoai/jazmo/.local/etc/gcloud', 'DOCKER_CONFIG': '/Users/mfairley/jazmoai/jazmo/.local/etc/docker'}), append_only_caches=FrozenDict({}), output_files=(), output_directories=(), timeout_seconds=-1, jdk_home=None, execution_slot_variable=None, concurrency_available=0, concurrency=None, cache_scope=<ProcessCacheScope.SUCCESSFUL: 'successful'>, remote_cache_speculation_delay_millis=0, attempt=0), run_in_workspace=False, forward_signals_to_process=True, restartable=False, keep_sandboxes=<KeepSandboxes.never: 'never'>)
Something to do with buildx running a separate process without the environment variables maybe?
Also, the thing that needs this is the docker_environment to build the pex binary, so maybe something to do with what docker_environment gets access when it needs to pull an image
w
Ah interesting. There are so many onion layers, that env vars can get stuck along the way - if they're not piped all the way through. I'll see if I have a private docker service somewhere I can upload to.. Or, is there something I can run on my machine to emulate a docker repo?
v
Yeah, looks like env vars are getting lost somewhere. Don't know about emulating a docker repo but it's pretty easy to set up a google cloud one. Also, funnily enough this issue you were working on before would have prevented this in my case (although wouldn't fix the root cause) https://github.com/pantsbuild/pants/issues/17714 because it could circumvent pulling from a remote repository
w
Yeah, I wiped the computer that had that fix. It was super trivial to do. I actually hate docker and podman, so I almost never use them - but as I've installed Fedora atomic on one of my computers, I always have access to it now (and I recently used rootless docker for something else), so I'm finally in a position to look at some of these docker issues again
😂 1
v
Much appreciated!
w
I'll see if I can make a repro with a private service when I get home tonight. I've only pushed to public docker hubs otherwise
👍 1
I setup a private registry last night, but I forgot to ask - can you provide an anonymized BUILD target and pants.toml so I can keep fidelity with your example?
v
Sure, I will send you one
👍 1
w
This turned into a bit of a set of tangents of things/maybe bugs I need to look into. I found interesting behaviour for certain functionality on atomic OSes (though, this is more just documentation/ergonomics), then semi-functional Podman behaviour, and then Pants sometimes using the command line calls - other times using Bollard’s API (for different flows). I'm going to re-try this with rootless docker, and see what happens - but a lot of the discrepancy actually came from Docker Environments (should be named Container Environments, probably), rather than using Pants + containers themselves.
👍 1
Should note that this turned into a bit of a cluster, specifically due to my atomic fedora, but what I wrote above summarizes it a bit (rootless docker isn't even correctly installing). I'll need to whip up another non-container environment to test this "correctly", but there is also a fair amount of work to get Podman (and eventually Apple Containers) running correctly here too - just so everyone is on an equalish playing field
👍 1
6 Views
Open in Slack
PreviousNext
Powered by