<#2312 Why does creating a virtualenv for torch ta...
# github-notifications
c
#2312 Why does creating a virtualenv for torch take about 7 seconds? Issue created by zmanji This is a question about pex's performance when creating virtualenvs. The example below is extracted from a Dockerfile where I am trying to understand if I can speed up the slowest step. I created a lockfile for torch by running:
Copy code
PEX_SCRIPT=pex3 pex lock create -v --resolver-version pip-2020-resolver --pip-version 22.3 --no-build -o torch.lock --platform 'linux_x86_64-cp-311-cp311' --platform 'macosx_11_0_arm64-cp-311-cp311' --platform 'linux_x86_64-cp-310-cp310' 'torch==2.1.2'
Then later I create venv by running:
Copy code
PEX_SCRIPT='pex3' pex venv create -v --force -d ./tenv  --no-build --lock torch.lock 'torch'
This takes about 7 seconds on my machine even when all of the wheels have been downloaded and extracted. The output shows: ``` PEX_SCRIPT='pex3' pex venv create -v --force -d ./tenv --no-build --lock torch.lock 'torch' pex: Using cached artifact at /home/zmanji/.pex/downloads/57dbda9b35157b05fb3e58ee91448612eb674172fab98ee235ccb0b5bee19a1c for FileArtifact(url='https://files.pythonhosted.org/packages/81/54/84d42a0bee35edba99dee7b59a8d4970eccdd44b99fe728ed912106fc781/filelock-3.13.1-py3-none-any.whl', fingerprint=Fingerprint(algorithm='sha256', hash='57dbda9b35157b05fb3e58ee91448612eb674172fab98ee235ccb0b5bee19a1c'), verified=False, filename='filelock-3.13.1-py3-none-any.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/3a871edd6c02dae77ad810335c0833391c1a4ce49af21ea8cf0f6a5d2096eea8 for FileArtifact(url='https://files.pythonhosted.org/packages/03/f1/13137340776dd5d5bcfd2574c9c6dfcc7618285035cd77240496e5c1a79b/torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='3a871edd6c02dae77ad810335c0833391c1a4ce49af21ea8cf0f6a5d2096eea8'), verified=False, filename='torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/d800d87f72189a745fa3d6b033b9dc4a34ad069f60ca60b943a63599f5501960 for FileArtifact(url='https://files.pythonhosted.org/packages/70/25/fab23259a52ece5670dcb8452e1af34b89e6135ecc17cd4b54b4b479eac6/fsspec-2023.12.2-py3-none-any.whl', fingerprint=Fingerprint(algorithm='sha256', hash='d800d87f72189a745fa3d6b033b9dc4a34ad069f60ca60b943a63599f5501960'), verified=False, filename='fsspec-2023.12.2-py3-none-any.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61 for FileArtifact(url='https://files.pythonhosted.org/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl', fingerprint=Fingerprint(algorithm='sha256', hash='6088930bfe239f0e6710546ab9c19c9ef35e29792895fed6e6e31a023a182a61'), verified=False, filename='Jinja2-3.1.2-py3-none-any.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/f18c69adc97877c42332c170849c96cefa91881c99a7cb3e95b7c659ebdc1ec2 for FileArtifact(url='https://files.pythonhosted.org/packages/d5/f0/8fbc882ca80cf077f1b246c0e3c3465f7f415439bdea6b899f6b19f61f70/networkx-3.2.1-py3-none-any.whl', fingerprint=Fingerprint(algorithm='sha256', hash='f18c69adc97877c42332c170849c96cefa91881c99a7cb3e95b7c659ebdc1ec2'), verified=False, filename='networkx-3.2.1-py3-none-any.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/ee53ccca76a6fc08fb9701aa95b6ceb242cdaab118c3bb152af4e579af792728 for FileArtifact(url='https://files.pythonhosted.org/packages/37/6d/121efd7382d5b0284239f4ab1fc1590d86d34ed4a4a2fdb13b30ca8e5740/nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='ee53ccca76a6fc08fb9701aa95b6ceb242cdaab118c3bb152af4e579af792728'), verified=False, filename='nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/339b385f50c309763ca65456ec75e17bbefcbbf2893f462cb8b90584cd27a1c2 for FileArtifact(url='https://files.pythonhosted.org/packages/b6/9f/c64c03f49d6fbc56196664d05dba14e3a561038a81a638eeb47f4d4cfd48/nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='339b385f50c309763ca65456ec75e17bbefcbbf2893f462cb8b90584cd27a1c2'), verified=False, filename='nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/794e3948a1aa71fd817c3775866943936774d1c14e7628c74f6f7417224cdf56 for FileArtifact(url='https://files.pythonhosted.org/packages/86/94/eb540db023ce1d162e7bea9f8f5aa781d57c65aed513c33ee9a5123ead4d/nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='794e3948a1aa71fd817c3775866943936774d1c14e7628c74f6f7417224cdf56'), verified=False, filename='nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/5ccb288774fdfb07a7e7025ffec286971c06d8d7b4fb162525334616d7629ff9 for FileArtifact(url='https://files.pythonhosted.org/packages/ff/74/a2e2be7fb83aaedec84f391f082cf765dfb635e7caa9b49065f73e4835d8/nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='5ccb288774fdfb07a7e7025ffec286971c06d8d7b4fb162525334616d7629ff9'), verified=False, filename='nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/e54fde3983165c624cb79254ae9818a456eb6e87a7fd4d56a2352c24ee542d7e for FileArtifact(url='https://files.pythonhosted.org/packages/7e/00/6b218edd739ecfc60524e585ba8e6b00554dd908de2c9c66c1af3e44e18d/nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='e54fde3983165c624cb79254ae9818a456eb6e87a7fd4d56a2352c24ee542d7e'), verified=False, filename='nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/6e258468ddf5796e25f1dc591a31029fa317d97a0a94ed93468fc86301d61e40 for FileArtifact(url='https://files.pythonhosted.org/packages/eb/d5/c68b1d2cdfcc59e72e8a5949a37ddb22ae6cade80cd4a57a84d4c8b55472/nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='6e258468ddf5796e25f1dc591a31029fa317d97a0a94ed93468fc86301d61e40'), verified=False, filename='nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/f3b50f42cf363f86ab21f720998517a659a48131e8d538dc02f8768237bd884c for FileArtifact(url='https://files.pythonhosted.org/packages/65/5b/cfaeebf25cd9fdec14338ccb16f6b2c4c7fa9163aefcf057d86b9cc248bb/nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='f3b50f42cf363f86ab21f720998517a659a48131e8d538dc02f8768237bd884c'), verified=False, filename='nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/8a7ec542f0412294b15072fa7dab71d31334014a69f953004ea7a118206fe0dd for FileArtifact(url='https://files.pythonhosted.org/packages/bc/1d/8de1e5c67099015c834315e333911273a8c6aaba78923dd1d1e25fc5f217/nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='8a7ec542f0412294b15072fa7dab71d31334014a69f953004ea7a118206fe0dd'), verified=False, filename='nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/9d264c5036dde4e64f1de8c50ae753237c12e0b1348738169cd0f8a536c0e1e0 for FileArtifact(url='https://files.pythonhosted.org/packages/44/31/4890b1c9abc496303412947fc7dcea3d14861720642b49e8ceed89636705/nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl', fingerprint=Fingerprint(algorithm='sha256', hash='9d264c5036dde4e64f1de8c50ae753237c12e0b1348738169cd0f8a536c0e1e0'), verified=False, filename='nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl') pex: Using cached artifact at /home/zmanji/.pex/downloads/… pantsbuild/pex