Hey! I am having a weird issue with the Docker env...
# general
f
Hey! I am having a weird issue with the Docker environment on OSX. Some times my tests take a really long time, and in the end I am just served with this weird error message:
Copy code
15:23:31.86 [INFO] Completed: Building pytest_runner.pex
15:23:31.87 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

IntrinsicError: Failed to start Docker container `510f97a829fa18c76691db4e556d07e31e16c55eea3aee870305a2779346ef6b` for image `sha256:e4bf78b64f77acd1f2201a360900c645575cd5aefdf46d4cc5025194651f30ac`: RequestTimeoutError
The docker container that pantsd uses (?) to avoid having to reinitialize the environment every time seems unresponsive. I can't kill it with for example docker kill:
Copy code
(requirements.pex) peder(feature/azure-devops-api)$ docker kill cranky_davinci
Error response from daemon: Cannot kill container: cranky_davinci: tried to kill container, but did not receive an exit event
(requirements.pex) peder(feature/azure-devops-api)$
Nothing works at this point, so I try to restart docker, but then Pants complains (rightfully so), that its docker container is missing:
Copy code
Failed to create Docker execution in container: DockerResponseServerError { status_code: 409, message: "Container 78d6a27f5e7c47ac676b0a42923670b049cc86d82c274a8e47b2a1006ddf9844 is not running" }
My questions are: • Why would the container become stuck in this weird state? This might be due to my environment, I sometimes have the wrong docker context set which causes docker to fail out for Pants some times • How would I let pantsd know to recreate the container instead of just complaining about it being missing?
Sorry about this information maybe being a little sparse, I was really unsure about how to forexample inspect the docker container when it is unresponsive in order to help you guys out. Let me know if there is any info you need to investigate this.
Something that always works is to delete the
.cache
,
.pants.d
and
.pids
folders, but that of course means that I have to wait a really long time. Is that the only way to fix this?
h
Hi, sorry for the trouble. Are you able to create a small example repo that reproduces this problem?
b
The second point is https://github.com/pantsbuild/pants/issues/18209 . The workaround I found is to kill pantsd only:
pkill -f pantsd
(This doesn’t help with the first/fundamental issue, though)
1
f
Hey Benjy! Yeah sure, I'll try my best, thank you for reaching out.
Sorry, really struggling to reproduce this. Letting you know when I've had some more time to work on this. In the mean time, Thanks Huon for your fix.
A minimal reprex with just some tests and the exact same config as our "actual" monorepo did not reproduce the issue.