#2292 Pex zip-creation takes a very long time for `torch>=2` Issue created by tgolsson Hey! Not sure if actionable, but maybe there's something here that can be done. I was investigating another issue today and ended up seeing a very slow Pants package step ~5 minutes. The issue reproduces with the simple command line

pex -vvv torch>=2 -o t2.2.pex

. This takes ~280 seconds on my machine, of which ~210-220 is spent purely in the zip step:

<snip>
pex: Building pex: 70298.8ms
pex:   Adding distributions from pexes: : 0.1ms
pex:   Resolving distributions for requirements: torch: 70294.7ms
pex:     Resolving requirements.: 70294.6ms
pex:       Resolving for:
  /usr/bin/python3.10: 55574.5ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='torch', processed_text='torch', source='<string>', start_line=1, end_line=1), requirement=Requirement(name='torch', url=None, extras=frozenset(), specifier=<SpecifierSet('')>, marker=None), editable=False): 0.1ms
pex:       Installing 22 distributions: 9352.3ms
pex:       Checking install: 2.7ms
pex:   Configuring PEX dependencies: 3.4ms
pex: Zipping PEX file.: 213135.5ms

This turns out to a 2.5 GB pex, which admittedly is on the fat side. Unzipping this beast takes ~30 seconds, and zipping it with regular

zip

takes ~230 seconds.

zip -1

takes ~100 seconds and adds ~10% to the size.

zip -0

takes 12 seconds but doubles the size. Seeing as compression seems to add the majority of the runtime, I did a very quick hack (outside of pex) where I move the compress step to a process pool (since it's CPU-heavy). With that, I get ~30 seconds at level 1, or about ~60 seconds on level 6. So 3-4x speed increase. It may be able to push this a bit higher by playing with ordering. I also played around with the store-only-by-suffix capabilities, but it seems like the .so's make up the bulk of both the compression potential and time: only compressing text-like files gives a ~4.3 GB zip in 20 seconds. With all that said, I'm mostly curious if this is something that has been discussed elsewhere (found nothing while searching), and what kind of solution might be palatable relative to the gains that can be made. I'm willing to contribute something based on the work I've done so far, or investigate other suggested approaches. pantsbuild/pex