Wanted to share a story about some issues we ran into with Pants and (primarily)

pex

. We recently integrated

pants

into a codebase and pushed some changes to production (we didn't catch these issues in our deployed development environment). In our

Dockerfile

we naively ran

pants package ::

and then just utilized the

pex

files generated without doing anything else. We have an

Argo Workflow

that spins up a bunch of pods (about 10 at launch and then ~ 100 more) and executes commands. We ended up with a lot of these pods stuck in "pod initializing" and a workflow that used to take 15 minutes ended up taking about an hour. Searching through metrics on what changed I found that our use of "ephemeral pod storage" sky rocketed. I downloaded the Docker images associated with each release and noticed when I ran the pex command and checked via

docker container ls -s

I was able to confirm that

pex

was using about 500MB of storage rather than 100B (what it was before the change). After following the instructions about [optimizing Docker deploys](https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/) I was able to get disk usage down. After deploying these changes, the workflow is now running smoothly. To be honest I don't understand the connection between ephemeral pod storage and pods failing to spin up (Argo Workflows was failing to spin up the

wait

container even after

main

had ran) I'm glad that blog article exists but looking at

pex

documentation I couldn't find anything about

PEX_TOOLS

or unpacking into a

venv

other than in the recipes area. Am I looking in the wrong place?

The Pants blog post and the Pex recipes were the correct and only places to look.

Thanks for sharing! We can look at how to make this information more easily available in the docs.