https://pantsbuild.org/ logo
#general
Title
# general
s

silly-queen-7197

02/27/2023, 9:27 PM
Wanted to share a story about some issues we ran into with Pants and (primarily)
pex
. We recently integrated
pants
into a codebase and pushed some changes to production (we didn't catch these issues in our deployed development environment). In our
Dockerfile
we naively ran
pants package ::
and then just utilized the
pex
files generated without doing anything else. We have an
Argo Workflow
that spins up a bunch of pods (about 10 at launch and then ~ 100 more) and executes commands. We ended up with a lot of these pods stuck in "pod initializing" and a workflow that used to take 15 minutes ended up taking about an hour. Searching through metrics on what changed I found that our use of "ephemeral pod storage" sky rocketed. I downloaded the Docker images associated with each release and noticed when I ran the pex command and checked via
docker container ls -s
I was able to confirm that
pex
was using about 500MB of storage rather than 100B (what it was before the change). After following the instructions about [optimizing Docker deploys](https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/) I was able to get disk usage down. After deploying these changes, the workflow is now running smoothly. To be honest I don't understand the connection between ephemeral pod storage and pods failing to spin up (Argo Workflows was failing to spin up the
wait
container even after
main
had ran) I'm glad that blog article exists but looking at
pex
documentation I couldn't find anything about
PEX_TOOLS
or unpacking into a
venv
other than in the recipes area. Am I looking in the wrong place?
e

enough-analyst-54434

02/27/2023, 9:32 PM
The Pants blog post and the Pex recipes were the correct and only places to look.
h

happy-kitchen-89482

02/27/2023, 10:39 PM
Thanks for sharing! We can look at how to make this information more easily available in the docs.
1
👍🏽 1
2 Views