https://pantsbuild.org/ logo
Join Slack
Powered by
:wave: we recently moved some apps to Pants/PEX, a...
# general
c
đź‘‹ we recently moved some apps to Pants/PEX, and we noticed a big increase in the container's memory usage (from ~300MB to ~1.5GB). We found out this was caused by linux caching after unzipping the file before running the app. Did anybody else experience the same issue? and if so, how did you solve it?
we have a workaround where we force pex to unzip the app when building the docker image but I wonder if there's a better solution
h
v
Yes we tried venv and other modes but they had no effect
w
after the 
venv
mode creates the venv, it becomes “just a venv”
so while the zip file might have temporarily been pulled into memory, it should not be pinned there… the pages of the file might just be in the page cache, and eligible to be evicted?
what memory stat are you looking at?
e
What Stu says is correct. That said, though, you can always take any 
--venv
mode PEX file and run `PEX_TOOLS=1 ./my.pex venv create/venv/here`That would allow you to use the PEX to eliminate PEX in a Dockerfile RUN step. Once you do so, there will be a 
create/venv/here/pex
file that can be executed just as if it were the original PEX file.
Concretely:
Copy code
COPY my.pex /my.pex
RUN PEX_TOOLS=1 /my.pex venv /my && \
    rm /my.pex
ENTRYPOINT ["/my/pex"]
In that image, 
/my
will be a standard venv and 
/my/pex
will be a script at its root (not in 
/my/bin/
) that runs just like the original 
/my.pex
file would have. This mode of container setup avoids the startup overhead of the 
/my.pex
--venv
PEX file turning itself into a venv automatically on 1st run; i.e.: You turn venv creation into an image build time step instead of a just-in-time runtime step.
🙌 1
a
The issue, btw, is mostly a monitoring one. The container in ECS (and generally if you’re looking at docker stats) shows up as using more memory and it’s hard to tell what’s the actual usage, but it also might need to take other things out of page cache to load this. Also, logically, it makes no sense to unzip the pex at first run, because every run will be the first run in a container. That solution looks sensible, though, and it’ll eliminate the duplication of having both a pex file and the extracted version
e
It will also run faster than an unzipped PEX file. Those still have PEX sys.path scrubbing runtime overhead that venvs do not.
It depends but this will save you O(100ms) in general.
c
Thank you all for the help 🙏
❤️ 1
3 Views
Open in Slack
PreviousNext
Powered by