What can I do if I'm trying package my PyTorch app with PEX but the resulting .zip ends up containing all the .so files for CUDA? They legitimately take up a lot of space. I was hoping that there could be a way to make PEX maybe not include those and let PyTorch instead rely on the system distribution of CUDA.

There's the

--exclude

option for pex which you could try adding to https://www.pantsbuild.org/dev/reference/targets/pex_binaries#extra_build_args.