I'm having really long (3-4 minutes) build times for requirements.pex lately alongside needing to rebuild it regularly despite no dependency changes; does anyone have any debugging suggestions?

Hmm, you shouldn’t need to rebuild it if there are no changes to the input requirements, unless you’ve been nuking the cache?

I am not

Where is that 3-4 minutes being spent? One way to find out is to run with

--keep-sandboxes=always

, look for a suspicious process (in this case probably an invocation of Pex) and then cd into the sandbox and look at/execute

__run.sh

So this is the process that seems to taking up all the time:

$ cat /tmp/pants-sandbox-nDex48/__run.sh
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /tmp/pants-sandbox-nDex48
env -i CPPFLAGS= LDFLAGS= PATH=$'/home/kuza/bin:/home/kuza/.local/bin:/home/kuza/google-cloud-sdk/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/kuza/.cargo/bin:/home/kuza/apache-maven-3.9.9/bin:/opt/gradle/gradle-7.3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Tailscale/:/mnt/c/Program Files/Git/cmd:/mnt/c/Docker/resources/bin:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Users/kuza5/AppData/Local/Programs/cursor/resources/app/bin:/mnt/c/Users/kuza5/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/kuza5/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Program Files/Git/bin:/snap/bin:/usr/lib/jvm/java-17-oracle/bin:/usr/lib/jvm/java-17-oracle/db/bin:/home/kuza/bin/' PEX_IGNORE_RCFILES=true PEX_PYTHON=/home/kuza/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.23.0a0/bin/python3.9 PEX_ROOT=.cache/pex_root /home/kuza/.cache/nce/68f5608a60df9b97aab453d453817a4ded400d1d8ec7ede7ec14bcac83421a7b/bindings/venvs/2.23.0a0/bin/python3.9 ./pex --tmpdir .tmp --jobs 5 --no-emit-warnings --pip-version latest --python-path $'/home/kuza/bin:/home/kuza/.local/bin:/home/kuza/google-cloud-sdk/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/kuza/.cargo/bin:/home/kuza/apache-maven-3.9.9/bin:/opt/gradle/gradle-7.3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Tailscale/:/mnt/c/Program Files/Git/cmd:/mnt/c/Docker/resources/bin:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Users/kuza5/AppData/Local/Programs/cursor/resources/app/bin:/mnt/c/Users/kuza5/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/kuza5/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Program Files/Git/bin:/snap/bin:/usr/lib/jvm/java-17-oracle/bin:/usr/lib/jvm/java-17-oracle/db/bin:/home/kuza/bin/' --output-file viz/dash_app.pex --emit-warnings $'--check=warn' --requirements-pex local_dists.pex --interpreter-constraint $'CPython<3.12,>=3.11' --entry-point viz.dash_app $'--sources-directory=source_files' $'absl-py==2.1.0; python_version >= "3.7"' $'aiohttp==3.10.6; python_version >= "3.8"' $'async-timeout==4.0.3; python_version >= "3.7"' $'asyncio==3.4.3' $'boto3==1.35.23; python_version >= "3.8"' $'botocore-stubs==1.35.29; python_version >= "3.8"' $'botocore==1.35.23; python_version >= "3.8"' $'cryptography==43.0.1; python_version >= "3.7"' $'dash-bootstrap-components==1.6.0; python_version >= "3.8" and python_version < "4"' $'dash-daq==0.5.0' $'dash==2.18.1; python_version >= "3.8"' $'httpx[http2]==0.27.2; python_version >= "3.8"' $'numpy==2.1.1; python_version >= "3.10"' $'opentelemetry-api==1.25.0; python_version >= "3.8"' $'opentelemetry-distro==0.46b0; python_version >= "3.8"' $'opentelemetry-exporter-otlp==1.25.0; python_version >= "3.8"' $'opentelemetry-instrumentation-aiohttp-client==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-fastapi==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-httpx==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-logging==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-pymysql==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-redis==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-requests==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-sqlalchemy==0.46b0; python_version >= "3.8"' $'opentelemetry-instrumentation-system-metrics==0.46b0; python_version >= "3.8"' $'opentelemetry-sdk==1.25.0; python_version >= "3.8"' $'opentelemetry-semantic-conventions==0.46b0; python_version >= "3.8"' $'pandarallel==1.6.5; python_version >= "3.7"' $'pandas-stubs==2.2.2.240909; python_version >= "3.10"' $'pandas==2.2.3; python_version >= "3.9"' $'plotly-resampler==0.10.0; python_full_version >= "3.7.1" and python_full_version < "4.0.0"' $'plotly==5.24.1; python_version >= "3.8"' $'polars==1.8.2; python_version >= "3.8"' $'portalocker==2.10.1; python_version >= "3.8"' $'protobuf==4.25.5; python_version >= "3.8"' $'pyarrow==17.0.0; python_version >= "3.8"' $'pydantic[email]==2.9.2; python_version >= "3.8"' $'pyjwt==2.9.0; python_version >= "3.8"' $'python-dateutil==2.9.0.post0; python_version >= "2.7" and python_version not in "3.0, 3.1, 3.2, 3.3"' $'redis==5.1.2.1; python_version >= "3.8"' $'s3fs==2024.9.0; python_version >= "3.8"' $'snowflake-connector-python==3.12.2; python_version >= "3.8"' $'splitio-client[cpphash]==9.6.0' $'sqlglot==25.23.2; python_version >= "3.7"' $'tqdm==4.66.5; python_version >= "3.7"' $'types-protobuf==4.25.0.20240417; python_version >= "3.8"' $'types-python-dateutil==2.9.0.20240906; python_version >= "3.8"' $'types-redis==4.6.0.20240903; python_version >= "3.8"' $'xgboost==2.1.1; python_version >= "3.8"' --lock 3rdparty/python/default.lock --no-pypi $'--index=<https://pypi.org/simple/>' $'--index=<https://pypi.fury.io/kuza55/>' --manylinux manylinux2014 --layout zipapp

Ah, yes, building large zip files can easily take a bunch of time