bitter-ability-32190
06/29/2022, 9:18 PMlayout="loose"
was not the tickethappy-kitchen-89482
06/29/2022, 9:30 PMbitter-ability-32190
06/29/2022, 9:30 PMwitty-crayon-22786
06/29/2022, 9:30 PMhappy-kitchen-89482
06/29/2022, 9:30 PMwitty-crayon-22786
06/29/2022, 9:30 PMhappy-kitchen-89482
06/29/2022, 9:31 PMwitty-crayon-22786
06/29/2022, 9:31 PMcareful-address-89803
06/29/2022, 9:32 PMls
). We solved it by sharding it into subfolders based on filename hash (I think the last 2 hexdigits, so "00" through "FF"). Though it sounds like solutions like that are off the tablewitty-crayon-22786
06/29/2022, 9:32 PMbitter-ability-32190
06/29/2022, 9:33 PMwitty-crayon-22786
06/29/2022, 9:35 PMcareful-address-89803
06/29/2022, 9:36 PMwitty-crayon-22786
06/29/2022, 9:36 PMopen
thembitter-ability-32190
06/29/2022, 9:37 PMwitty-crayon-22786
06/29/2022, 9:39 PMbitter-ability-32190
06/29/2022, 9:40 PMwitty-crayon-22786
06/29/2022, 9:58 PMenough-analyst-54434
06/30/2022, 3:04 AMbitter-ability-32190
06/30/2022, 5:38 AMPEX_TOOLS=1 python3.8 /bin/app.pex venv --scope=deps --compile /app
calls are the slow ones
⦠I don't have much imagination on debugging this, but I suspect bundling a PEX with >100k firstparty sources is slow to unpack?
⢠Building the PEX without the files, and having docker place them in the image seems "fast" by comparisonenough-analyst-54434
07/01/2022, 12:26 AM--scope=deps
but mention >100k 1st party sources. 1st party != deps so they should be unrelated. I guess I'm not tracking your thought process combining those two elements there.bitter-ability-32190
07/01/2022, 12:29 AMenough-analyst-54434
07/01/2022, 12:36 AM# Create 100k:
$ mkdir big
$ man ls > big/seed
$ for i in `seq 1 100000`; do cp big/seed big/$i; done
$ du -sh big/
782M big/
# Native zip store / unpack 100k no compression:
$ time zip -qr --compression-method store big.stored.zip big/
real 0m2.187s
user 0m1.552s
sys 0m0.624s
$ time unzip -q -d stored big.stored.zip
real 0m3.427s
user 0m2.661s
sys 0m0.673s
# Native zip store / unpack 100k default compression:
$ time zip -qr big.default.zip big/
real 0m12.674s
user 0m12.083s
sys 0m0.576s
time unzip -q -d default big.default.zip
real 0m3.757s
user 0m3.167s
sys 0m0.547s
# Create PEX / unpack PEX default compression 100k 1st party:
$ time pex -D big -o big.default.pex
real 0m18.790s
user 0m17.191s
sys 0m1.545s
$ time pex-tools big.default.pex venv /tmp/big.default.pex.venv
real 0m14.282s
user 0m8.930s
sys 0m5.193s
# Create PEX / unpack PEX no compression 100k 1st party:
$ time pex -D big --no-compress -o big.stored.pex
real 0m6.879s
user 0m5.129s
sys 0m1.713s
$ time pex-tools big.stored.pex venv /tmp/big.stored.pex.venv
real 0m5.205s
user 0m3.622s
sys 0m1.539s
bitter-ability-32190
07/01/2022, 12:38 AMenough-analyst-54434
07/01/2022, 12:40 AMbitter-ability-32190
07/01/2022, 12:41 AMenough-analyst-54434
07/01/2022, 12:41 AMbitter-ability-32190
07/01/2022, 12:42 AMenough-analyst-54434
07/01/2022, 12:42 AM