https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all, I am trying to figure out why the same d...
# general
g
Hey all, I am trying to figure out why the same django app running in docker is almost using double the memory when running using pants compared to running my old manual docker file. I am using the template example-django in my pants project. and I have gunicorn configured with same params in both cases.
Copy code
workers = 1
threads = 8
timeout = 0
Dockerfile-old.dockerfile
Dockerfile-old.dockerfile
Dockerfile-pants.dockerfile
Dockerfile-pants.dockerfile
are there any gotchas with 
pex
files?
h
How are you building the pex? Have you tried building a venv-mode pex?
g
Almost identical to the example-django repo
BUILD.py
h
@bitter-ability-32190 you looked at pex-in-docker performance extensively, any thoughts?
g
no, will check
Actually also startup time almost more than doubled. This the same image deployed to cloud run
h
I don't anything about cloud run, is that running containers for you?
g
Yeah, just horz scalable and ephemeral container runner. Similar to cloud functions v2. EDIT: It optionally scales to 0
h
And that blog post I linked to above has examples on how to build and deploy venv pexes
So follow along with that and let us know here how that goes
b
We never looked at memory. Using a venv PEX already pre-installed should yield EXACTLY the same results as just making the venv yourself (assuming everything else is similar. Code l, packages, etc...)
g
I doing an experiment comparing all combinations, will report here
Need to figure out startup time next to decide the best version. For now, looks like 
venv-zipapp
is the middle ground between size and memory
e
@glamorous-nail-59544 what Josh said is true and so it looks like you must not be following the link Benjy pointed to (https://pex.readthedocs.io/en/v2.1.120/recipes.html?highlight=layout#pex-app-in-a-container) or Josh's blogs. Those both have you pre-build the venv in the container at container build time. That 1 time container build step should make packed, loose, zipapp all the same in terms of mem and timings.
Perhaps this is hard to do with Pants docker integration?
I don't know much about our Docker integration. I write manual docker files and build them outside of Pants.
g
I looked at the blog, haven't checked pex docs shared above yet. Will look at that now. Was comparing the available built in options. Might get away without the added complexity of the solution mentioned in pex docs.
e
Ok. Just know that the tradeoff is a few more lines of Dockerfile for lowest latency + memory usage all in one.
b
With the blog post it's even more lines/plumbing but focuses on incremental caches making it worth it
@enough-analyst-54434 our docker integration currently uses hand written docker files. So complete flexibility
g
I missed the blog post above, was looking at this. Working on getting the most optimal version now 😌
had some fun with detailed benchmarks
results.csv
c
@glamorous-nail-59544 Hi. Can I ask what version did you choose? We are using dockerfile-optimal-multi-stage-2pex but still has some memory issue and I am looking into it.
g
Untitled.dockerfile
@curved-farmer-66180 hey, I ended up using it as well, here is a sample dockerfile
Untitled.dockerfile
c
thanks! @glamorous-nail-59544 It is almost same as ours. I found the memory issue was from gunicorn configuration.
5 Views
Open in Slack
PreviousNext
Powered by