https://pantsbuild.org/ logo
Join Slack
Powered by
Hey folks - I'm trying to shake out the interactio...
# general
b
Hey folks - I'm trying to shake out the interaction between 
[docker].env_vars / tools
, and a docker context which uses a docker host via ssh. I'm able to see that the local process is spawned for docker, and PATH is set to a private .shims/bin. 
ssh
is defined within 
[docker].tools
, but I am getting 
docker: error during connect: Post "<http://docker.example.com/v1.24/containers/create>": exec: "ssh": executable file not found in $PATH.
The basic test (symlinking, and 
env -i
) works trivially.
With 
-ldebug --no-process-cleanup
, I was able to validate that the ssh binary is dropped into the 
.shims/bin
folder, and it is marked executable.
Copy code
env -i PATH=/private/var/folders/0j/xmkn5s9x4j9_9x1qfjz6g3t80000gn/T/process-executiongmM17s/.shims/bin ssh ubuntu@100.124.76.55 whoami
ubuntu
pants package
goal appears to "do the right thing"; appears to be a problem in the "run" goal. I can 
docker run
the generated image, and a 
docker images
does show the built image. I'm guessing that the 
run
goal doesn't leverage the 
tools
key - Is there a workaround for this?
h
Hi, sorry for the trouble. @curved-television-6568 is the author of the docker support and the main expert, but I think he's out for a bit. Can you post a trivial public repo that reproduces this? That would help me page in enough context to debug this.
b
I frustratingly can't - even though this is a personal project, employer currently prevents me from putting anything public.
I believe the minimal repro is really just a docker context using a docker-host over SSH. In my case I'm calling from an m1 mac to a linux host (such that I don't need to spin up the VM on the machine).
c
@happy-kitchen-89482 is right, I'm out for a while longer. I'll throw in one thing to check early. Just to confirm which version of Pants are you on? I think this issue may be relevant https://github.com/pantsbuild/pants/issues/15322
b
I'm on 2.12.0
h
That fix is in 2.12.0, so I guess it's not that. Grrr.
But it does really seem like that issue... hmmm
b
My gut feeling is that it's a difference between how the env is built for package vs run, but I haven't fully wound my way through it yet.
h
What does the relevant BUILD file look like?
b
I'll snag it in a bit. It's pretty barebones. Python sources, pex binary of the above, targeting macos arm64 and Linux x8664, and a docker image containing the pex, with a bare minimum copy/run docker file.
h
I wasn't able to reproduce this on 2.12 (or on main)
The ssh connection succeeds (and then errors because there's no docker binary on the remote host I was testing against, but that's another issue)
@blue-queen-76603 with 
-ldebug --no-process-cleanup
, what do you see in the 
__run.sh
for the PATH?
(obligatory note that 
__run.sh
is just a debugging convenience and not what Pants actually uses to run the process, but it is instructive)
b
I'll try to capture this again in a bit.
Copy code
jeremy@deskm1ni beeblyboop % cd /private/var/folders/0j/xmkn5s9x4j9_9x1qfjz6g3t80000gn/T/process-executiongAKnzM/
jeremy@deskm1ni process-executiongAKnzM % ls
__run.sh	beeblyboop
jeremy@deskm1ni process-executiongAKnzM % cat __run.sh 
#!/bin/bash
# This command line should execute the same process as pants did internally.
export DOCKER_CONFIG=/Users/jeremy/.docker DOCKER_CONTEXT=littlebits PATH=/private/var/folders/0j/xmkn5s9x4j9_9x1qfjz6g3t80000gn/T/process-executiongAKnzM/.shims/bin
cd /private/var/folders/0j/xmkn5s9x4j9_9x1qfjz6g3t80000gn/T/process-executiongAKnzM
/opt/homebrew/bin/docker build --tag $'<http://docker.jeremydk.com/beeblyboop/docker:latest|docker.jeremydk.com/beeblyboop/docker:latest>' --file beeblyboop/Dockerfile .
Copy code
jeremy@deskm1ni process-executiongAKnzM % ls -alh .shims/bin 
total 8
dr-xr-xr-x  3 jeremy  staff    96B Jul 29 12:15 .
dr-xr-xr-x  3 jeremy  staff    96B Jul 29 12:15 ..
-r-xr-xr-x  1 jeremy  staff    37B Jul 29 12:15 ssh
Copy code
jeremy@deskm1ni process-executiongAKnzM % cat .shims/bin/ssh
#!/bin/bash
exec "/usr/bin/ssh" "$@"
So the odd thing that I see is that the package goal itself is able to use ssh with the docker context - it's just the docker run step.
Which makes me wonder if that's the chroot for package and not run
Sent you a gist in DMs. I'm not sure I'm seeing the sandbox for the 
docker run
step at all, only the 
docker build
step.
h
Ah, right, in 2.12 
run
used a different mechanism for preserving the sandbox
Try 
--no-run-cleanup
b
the binary chroot is empty.
Copy code
17:06:49.22 [INFO] Preserving running binary chroot /Users/jeremy/workspace/beeblyboop/.pants.d/tmpujcgz_5e
docker: error during connect: Post "<http://docker.example.com/v1.24/containers/create>": exec: "ssh": executable file not found in $PATH.
See 'docker run --help'.
17:06:49.24 [DEBUG] Completed: `run` goal
17:06:49.24 [DEBUG] computed 1 nodes in 3.621262 seconds. there are 625 total nodes.
jeremy@deskm1ni beeblyboop % ls -al .pants.d/tmpujcgz_5e 
total 0
drwx------   2 jeremy  staff   64 Jul 29 17:06 .
drwxr-xr-x  18 jeremy  staff  576 Jul 29 17:06 ..
and from there - 
PANTS_DOCKER_ENV_VARS=PATH=$PATH ./pants run ...
does also work
although obviously heavyhanded
h
Huh. We did change some things around how run works in 2.13, maybe try upgrading?
This is a tad mysterious
b
Repros on 2.13.0rc0 for me.
h
Urgh
Are you able to share a full repro privately? I.e., a complete simple repo that demonstrates the issue?
Maybe one of the blanks I filled in was the issue
b
I should have time to put something together today.
What pants version would you like me to target with it?
h
If it reproduces on 
2.14.0.dev3
then that
b
Sounds good, will see what I can do.
Just added you as a contributor onto the minimal repro case, and pulled in a minimal docker config + context. You'll need a remote host (in my case, running ubuntu 22.04 LTS) with public key SSH auth which has docker running.
I really appreciate looking into this - I know this is probably not a common configuration/setup, but I find it incredibly valuable when doing development on a macos client machine. (macos client with only the docker CLI, a linux box running Docker, tailscale for connectivity between)
h
Thanks, looking now
Now I can reproduce this, so that's something!
b
Awesome, glad the minimal repo helps.
h
You found a proper bug! I've got a fix, will send it out shortly.
❤️ 1
Turned out to be a very simple fix: https://github.com/pantsbuild/pants/pull/16385
Thanks @blue-queen-76603 for the report and the repro
b
Thanks for diving in and getting a fix out.
4 Views
Open in Slack
PreviousNext
Powered by