I m seeing very high startup times on a certain docker image

Question

I m seeing very high startup times on a certain docker image which happens to be running a PEX binary thats ~1 5GB in size and contains at least 1 GB worth of a dir containing 100k files Any tips on making this less more startup friendly slightly smiling face

bitter-ability-32190 · Answer

`layout= loose ` was not the ticket

happy-kitchen-89482 · Answer

Is this venv mode

bitter-ability-32190 · Answer

I might run the app in my dockerfile so it s uploaded unpacked

bitter-ability-32190 · Answer

Yes venv mode

witty-crayon-22786 · Answer

running as a true zipapp would avoid actually extracting the contents to disk first

happy-kitchen-89482 · Answer

Yeah so the one time creation of the venv is supposed to amortize over multiple invocations

witty-crayon-22786 · Answer

but i don t think that that mode exists anymore

happy-kitchen-89482 · Answer

but in the docker world you often only have the one invocation per container

happy-kitchen-89482 · Answer

I think it does

happy-kitchen-89482 · Answer

If you don t set venv or loose

witty-crayon-22786 · Answer

< happy kitchen 89482> modified zipapp

witty-crayon-22786 · Answer

pyoxidizer gets you the moral equivalent of a zipapp

careful-address-89803 · Answer

we had a problem with a dir with fewer files than that causing problems with basic file operations like `ls` We solved it by sharding it into subfolders based on filename hash I think the last 2 hexdigits so 00 through FF Though it sounds like solutions like that are off the table

witty-crayon-22786 · Answer

filesystems suck =

bitter-ability-32190 · Answer

As the build guy application changes like that are hard to request sob

bitter-ability-32190 · Answer

Also Stu we use dunder file all over so true zipapp is a tiny nonstarter sob

witty-crayon-22786 · Answer

you might be able to fiddle with those files such that they are appended to the pythonpath as a zipfile

careful-address-89803 · Answer

Oof I figured You might also be able to not include the files in the PEX and have them included in a separate stage of the docker build That way the wild fileio happens before the pex unzip

witty-crayon-22786 · Answer

< bitter ability 32190> but is this a case where you actually need these on the pythonpath

witty-crayon-22786 · Answer

if not you could put them next to the PEX itself in whichever format and then `open` them

bitter-ability-32190 · Answer

Aren t I unzipping them either way though We didn t solve the original issue

bitter-ability-32190 · Answer

I m thinking of just adding a stage to the build where I call my app with help upside down face

witty-crayon-22786 · Answer

if this is about incurring a one time cost so that future startups are fast then yea that s very reasonable

bitter-ability-32190 · Answer

Is there a magical command that does the spiritual equivalent to running the PEX such that it is just unpacked

bitter-ability-32190 · Answer

If only I could test this outside of docker by running the built PEX in one command joy wink thinking face

witty-crayon-22786 · Answer

if you build with embedded pex tools you can explicitly ask a PEX to extract to a venv

witty-crayon-22786 · Answer

but i don t know about seed your usual cached venv and then exit flag

witty-crayon-22786 · Answer

enough-analyst-54434 · Answer

Have you seen this <https pex readthedocs io en v2 1 93 recipes html pex app in a container|https pex readthedocs io en v2 1 93 recipes html pex app in a container>

enough-analyst-54434 · Answer

< witty crayon 22786> that mode ~never did what you thought it did For Pex 2 x true zipapp just ran user code from zip All 3rd party deps we re always extracted as the easy hedge against platform specific wheels with embedded ` so`s Those must be unzipped to load at all This goes all the way back to the intro of wheel support in Pex

bitter-ability-32190 · Answer

Yeah that reminds me we also have first party wheels with embedded so s

bitter-ability-32190 · Answer

I ll try your suggestion first and decouple the big dir if that doesn t work

bitter-ability-32190 · Answer

I think I m going to do both Globally use your link as a general docker container recipe for our pex binaries And then also make this binary exclude the dep and then in another stage for this container copy and unzip the fat dir

bitter-ability-32190 · Answer

From running Docker in the sandbox Transferring the PEX to the daemon is speedy I think `PEX TOOLS=1 python3 8 bin app pex venv scope=deps compile app` calls are the slow ones I don t have much imagination on debugging this but I suspect bundling a PEX with gt 100k firstparty sources is slow to unpack Building the PEX without the files and having docker place them in the image seems fast by comparison

enough-analyst-54434 · Answer

< bitter ability 32190> to your second point You pinpoint ` scope=deps` but mention gt 100k 1st party sources 1st party = deps so they should be unrelated I guess I m not tracking your thought process combining those two elements there

enough-analyst-54434 · Answer

FWIW most interesting Pex timings are revealed with PEX VERBOSE=1 or 2 higher gets noisy

enough-analyst-54434 · Answer

As far as packing and unpacking go it s the packing with default compress that can be slow I added no compress a while back but Pants does not expose a knob That can make packing considerably faster at the cost of size

bitter-ability-32190 · Answer

Ah I copied the first line that should be scope=srcs

enough-analyst-54434 · Answer

Ok What do you mean by slow then My experiment ``` Create 100k $ mkdir big $ man ls gt big seed $ for i in `seq 1 100000` do cp big seed big $i done $ du sh big 782M big Native zip store unpack 100k no compression $ time zip qr compression method store big stored zip big real 0m2 187s user 0m1 552s sys 0m0 624s $ time unzip q d stored big stored zip real 0m3 427s user 0m2 661s sys 0m0 673s Native zip store unpack 100k default compression $ time zip qr big default zip big real 0m12 674s user 0m12 083s sys 0m0 576s time unzip q d default big default zip real 0m3 757s user 0m3 167s sys 0m0 547s Create PEX unpack PEX default compression 100k 1st party $ time pex D big o big default pex real 0m18 790s user 0m17 191s sys 0m1 545s $ time pex tools big default pex venv tmp big default pex venv real 0m14 282s user 0m8 930s sys 0m5 193s Create PEX unpack PEX no compression 100k 1st party $ time pex D big no compress o big stored pex real 0m6 879s user 0m5 129s sys 0m1 713s $ time pex tools big stored pex venv tmp big stored pex venv real 0m5 205s user 0m3 622s sys 0m1 539s```

enough-analyst-54434 · Answer

I m sure my big is not matching your case but for starters of something we can share

bitter-ability-32190 · Answer

Startup first time was in the hundreds of seconds When we shifted that to using the unpack first I saw similar

bitter-ability-32190 · Answer

I can spend some time tomorrow timing FWIW We fixed it by using that multi stage build you linked tada and added a new stage which copied the sources That way when the code changes it doesn t need to recopy Overall a win

enough-analyst-54434 · Answer

Ok stepping back If you pre create the venv with compile then you re left with a native precompiled venv That should have the same startup time as a native venv Are you debating that or are you just concerned with the time taken to pre seed the venv in the Dockerfile build step

bitter-ability-32190 · Answer

Not debating Startup time when not seeded pre seed time when seeded

enough-analyst-54434 · Answer

Perhaps file some issues with more context tomorrow if I need to do anything I m still a bit lost if you re sorted or left wanting

bitter-ability-32190 · Answer

definitely sorted Multi stage build was the hot ticket for all our images using pex and an additional stage made this speedy

enough-analyst-54434 · Answer

OK great