https://pantsbuild.org/ logo
Join Slack
Powered by
Going to take notes in this thread. I was able to ...
# general
j
Going to take notes in this thread. I was able to reproduce this with the example repo (
pantsbuild/example-python
) on my ec2 box:
Copy code
buildkite-agent@raulcicd:~/builds/foo/example-python$ ps aux | grep [p]antsd
buildki+  3339  0.3  2.3 260008 46528 ?        Sl   13:54   0:00 pantsd [/var/lib/buildkite-agent/builds/foo/example-python]
buildkite-agent@raulcicd:~/builds/foo/example-python$ kill 3339
buildkite-agent@raulcicd:~/builds/foo/example-python$ ./pants -ldebug list ::
Scrubbed PYTHONPATH=/home/ubuntu/superfly from the environment.
13:57:19.52 [DEBUG] acquiring lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7f65b92f0470>
13:57:19.53 [DEBUG] purging metadata directory: /var/lib/buildkite-agent/builds/foo/example-python/.pids/pantsd
13:57:19.53 [DEBUG] Launching pantsd
13:57:19.53 [DEBUG] purging metadata directory: /var/lib/buildkite-agent/builds/foo/example-python/.pids/pantsd
13:57:19.53 [DEBUG] pantsd command is: PANTS_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/pants.gUms69/install/bin:/home/ubuntu/superfly:/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python36.zip:/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6:/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/lib-dynload:/usr/lib/python3.6:/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/lib/python3.6/site-packages /var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/bin/python /var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/bin/pants --pants-bin-name=./pants --pants-version=2.0.0 -ldebug list ::
13:57:20.23 [DEBUG] pantsd is running at pid 6001, pailgun port is 40237
13:57:20.23 [DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7f65b92f0470>
13:57:20.23 [DEBUG] connecting to pantsd on port 40237 (attempt 1/3)
Failed to launch child `/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/bin/pants`: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
buildkite-agent@raulcicd:~/builds/foo/example-python$ ps aux | grep [p]antsd
buildki+  6001 13.4  2.3 260008 47060 ?        Sl   13:57   0:00 pantsd [/var/lib/buildkite-agent/builds/foo/example-python]
Excuse me. I was not reading carefully and was confusing 
port
and 
PID
.
Is it correct to say that after 
[DEBUG] connecting to pantsd on port 40237 (attempt 1/3)
, my running 
pants
command successfully connected, passed work to 
pantsd
and then 
pantsd
responded with 
Failed to launch child...
?
Thanks to Stu who pointed me to the 
.pants.d/pantsd/pantsd.log
. I can see that the connection is indeed being made, but I cannot see what command 
pantsd
is being asked to run. From the error message on the client it is probably 
/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/bin/pants
.
Why can't the process 
pantsd
which is running under 
buildkite-agent
user execute this command?
I'm going to try and reproduce in Docker.
How to reproduce: * build dockerfile
Copy code
FROM ubuntu:18.04
RUN apt update && apt upgrade -y && apt install -y locales language-pack-en
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
RUN apt install -y curl python python3-dev git vim build-essential unzip tar
* 
docker run --rm -it $imageid
Copy code
# adduser cow
# su - cow
cow $ git clone <https://github.com/pantsbuild/example-python>
cow $ export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8
cow $ cd example-python
cow $ ./pants -ldebug version
If I exit and run 
./pants version
from the 
~cow/expample-python
repo, it also fails. BUT if I clone the repo from the root account, it works.
@enough-analyst-54434 I was able to reproduce the "Permission denied" problem using Docker. ☝đŸŊ
Copy code
root@ef680066facd:~# cd /var/log/example-python/
root@ef680066facd:/var/log/example-python# ./pants list ::
//:ansicolors
//:protobuf
//:requirements.txt
//:setuptools
//:translate
helloworld
helloworld:config
helloworld:config_file
helloworld:helloworld-awslambda
helloworld:helloworld_py2
helloworld/greet
helloworld/greet:tests
helloworld/greet_py2
helloworld/util
helloworld/util:config_loader_test_data
helloworld/util:dist
helloworld/util:tests
helloworld/util/proto
helloworld/util/proto:init
root@ef680066facd:/var/log/example-python# cd ~cow/example-python/
root@ef680066facd:/home/cow/example-python# ./pants list ::
Failed to launch child `/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36/bin/pants`: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
e
OK - great. Thanks @jolly-midnight-72759.
The issue is /dev/pts/0 inside the container. Thats the terminal device and its perms are:
Copy code
cow@0251e45e8282:~$ ls -l /dev/pts/0
crw--w---- 1 root tty 136, 0 Nov 10 22:13 /dev/pts/0
When pants connects to pantsd it passes that device name along and pantsd tries to open it so it can interact with pants for example in 
repl
. Its that file that the perm denied is about. I fixed by (as root) 
adduser cow tty && chmod g+r /dev/pts/0
.
j
fate kept me from making an issue. if one is needed, I can make it the next time fate brings me back to my keyboard. đŸŊī¸
e
Aha, this is all avoided if you add user creation to the image and run the image specifying or default to that user - docker then sets up 
-t
/ pty perms correctly.
With:
Copy code
FROM ubuntu:18.04
RUN apt update && apt upgrade -y && apt install -y locales language-pack-en
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
RUN apt install -y curl python python3-dev git vim build-essential unzip tar

RUN adduser --disabled-password --gecos "" cow
RUN apt install -y strace
USER cow
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
WORKDIR /home/cow
And: 
docker build -t raul:3.6 .
I find:
Copy code
docker run --rm -it raul:3.6 bash -c 'git clone <https://github.com/pantsbuild/example-python> && export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8 && cd example-python && ./pants version'
...
New virtual environment successfully created at /home/cow/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0_py36.
22:26:42.75 [INFO] initializing pantsd...
22:26:43.60 [INFO] pantsd initialized.
2.0.0
So I don't think this is Pants issue worthy. This is a generic issue using Docker the way you did with an interactive process (passing 
docker run -t
). Creating the user on the fly and `su`ing to them will leave you with a borked tty.
In terms of helping yourself going forward, I figured this out using 
strace -p <pantsd pid>
as root. That required running docker with additional flags: 
docker run --rm -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --security-opt apparmor=unconfined raul:3.6
Um, actually 
strace -f -p <pantsd pid>
- you need the -f to trace into the fork pantsd does to accept the pants client connection. Its in that fork that you see:
Copy code
root@5b6129f58b59:/# grep -i perm strace.pantsd-180.log.txt 
[pid   196] openat(AT_FDCWD, "/dev/pts/0", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
j
Excellent. I can use that to see why my ec2 and buildkite user are failing.
🌩ī¸ Same error: 
openat(AT_FDCWD, "/dev/pts/1", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
e
I don't believe you.
j
haha
e
So you used the new Dockerfile?
j
I am using my ec2
No Docker involved.
e
Oh - gotcha.
j
This is an Ubuntu issue
But I should be able to launch a daemon and use it
e
OK, "great". Great for me anyhow, It sounds like you're off and running with the tools / narrowing of scope you need to debug further?
j
yes. I will see what I can discover. The strace shows what is getting denied.
e
Cool.
j
I can run the buildkite agent with 
--no-pty
.
e
Excellent.
Open in Slack
PreviousNext
Powered by