silly-queen-719702/27/2023, 9:27 PM
. We recently integrated
into a codebase and pushed some changes to production (we didn't catch these issues in our deployed development environment). In our
we naively ran
and then just utilized the
pants package ::
files generated without doing anything else.
We have an
that spins up a bunch of pods (about 10 at launch and then ~ 100 more) and executes commands. We ended up with a lot of these pods stuck in "pod initializing" and a workflow that used to take 15 minutes ended up taking about an hour. Searching through metrics on what changed I found that our use of "ephemeral pod storage" sky rocketed.
I downloaded the Docker images associated with each release and noticed when I ran the pex command and checked via
I was able to confirm that
docker container ls -s
was using about 500MB of storage rather than 100B (what it was before the change). After following the instructions about [optimizing Docker deploys](https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/) I was able to get disk usage down. After deploying these changes, the workflow is now running smoothly.
To be honest I don't understand the connection between ephemeral pod storage and pods failing to spin up (Argo Workflows was failing to spin up the
container even after
I'm glad that blog article exists but looking at
documentation I couldn't find anything about
or unpacking into a
other than in the recipes area. Am I looking in the wrong place?
enough-analyst-5443402/27/2023, 9:32 PM
happy-kitchen-8948202/27/2023, 10:39 PM