https://pantsbuild.org/ logo
g

glamorous-nail-59544

01/13/2023, 12:18 PM
Hey all, I am trying to figure out why the same django app running in docker is almost using double the memory when running using pants compared to running my old manual docker file. I am using the template example-django in my pants project. and I have gunicorn configured with same params in both cases.
Copy code
workers = 1
threads = 8
timeout = 0
Dockerfile-old.dockerfile
Dockerfile-pants.dockerfile
are there any gotchas with
pex
files?
h

happy-kitchen-89482

01/13/2023, 12:24 PM
How are you building the pex? Have you tried building a venv-mode pex?
g

glamorous-nail-59544

01/13/2023, 12:25 PM
Almost identical to the example-django repo
h

happy-kitchen-89482

01/13/2023, 12:26 PM
@bitter-ability-32190 you looked at pex-in-docker performance extensively, any thoughts?
g

glamorous-nail-59544

01/13/2023, 12:27 PM
no, will check
Actually also startup time almost more than doubled. This the same image deployed to cloud run
h

happy-kitchen-89482

01/13/2023, 12:41 PM
I don't anything about cloud run, is that running containers for you?
g

glamorous-nail-59544

01/13/2023, 12:43 PM
Yeah, just horz scalable and ephemeral container runner. Similar to cloud functions v2. EDIT: It optionally scales to 0
And that blog post I linked to above has examples on how to build and deploy venv pexes
So follow along with that and let us know here how that goes
b

bitter-ability-32190

01/13/2023, 1:22 PM
We never looked at memory. Using a venv PEX already pre-installed should yield EXACTLY the same results as just making the venv yourself (assuming everything else is similar. Code l, packages, etc...)
g

glamorous-nail-59544

01/13/2023, 1:29 PM
I doing an experiment comparing all combinations, will report here
Need to figure out startup time next to decide the best version. For now, looks like
venv-zipapp
is the middle ground between size and memory
e

enough-analyst-54434

01/13/2023, 3:09 PM
@glamorous-nail-59544 what Josh said is true and so it looks like you must not be following the link Benjy pointed to (https://pex.readthedocs.io/en/v2.1.120/recipes.html?highlight=layout#pex-app-in-a-container) or Josh's blogs. Those both have you pre-build the venv in the container at container build time. That 1 time container build step should make packed, loose, zipapp all the same in terms of mem and timings.
Perhaps this is hard to do with Pants docker integration?
I don't know much about our Docker integration. I write manual docker files and build them outside of Pants.
g

glamorous-nail-59544

01/13/2023, 3:13 PM
I looked at the blog, haven't checked pex docs shared above yet. Will look at that now. Was comparing the available built in options. Might get away without the added complexity of the solution mentioned in pex docs.
e

enough-analyst-54434

01/13/2023, 3:34 PM
Ok. Just know that the tradeoff is a few more lines of Dockerfile for lowest latency + memory usage all in one.
b

bitter-ability-32190

01/13/2023, 3:45 PM
With the blog post it's even more lines/plumbing but focuses on incremental caches making it worth it
@enough-analyst-54434 our docker integration currently uses hand written docker files. So complete flexibility
g

glamorous-nail-59544

01/13/2023, 3:48 PM
I missed the blog post above, was looking at this. Working on getting the most optimal version now 😌
had some fun with detailed benchmarks
c

curved-farmer-66180

07/09/2023, 5:29 AM
@glamorous-nail-59544 Hi. Can I ask what version did you choose? We are using dockerfile-optimal-multi-stage-2pex but still has some memory issue and I am looking into it.
g

glamorous-nail-59544

07/09/2023, 7:20 PM
Untitled.dockerfile
@curved-farmer-66180 hey, I ended up using it as well, here is a sample dockerfile
c

curved-farmer-66180

07/11/2023, 10:34 PM
thanks! @glamorous-nail-59544 It is almost same as ours. I found the memory issue was from gunicorn configuration.
5 Views