https://pantsbuild.org/ logo
Join Slack
Powered by
I'm facing some very slow Python build times. Any ...
# general
n
I'm facing some very slow Python build times. Any time any dependency is changed the build takes forever, primarily on steps like 
Building 21 requirements for requirements.pex
which I think is building a pex isolated environment for unit tests and/or mypy runs. i think the main culprit is a few big libraries like PyTorch and torchvison but, for obvious reasons, I can't eliminate those. It's gotten so bad that I just added a single dependency and my CI is now timing out after 2 hours even though I'm caching the pants cache in my CI. Is there anything I can do to speed this up? I'm looking at options like maybe running with 
execution_mode = 'venv'
or 
run_against_entire_lockfile
 but I don't have a good enough understand of how Pants, pex, testing, pip, etc. all interact and since each experiment takes about 2 hours some advice would be greatly appreciated.
h
run_against_entire_lockfile
will prevent lockfile subsetting, which may help. I think that is definitely worth trying. The downside is that all tests will be invalidated on any lockfile change, but that may not be significant, especially in CI.
n
Thanks @happy-kitchen-89482. I just did a local run with 
run_against_entire_lockfile
and it was pretty fast - 9 minutes. It's running in CI now. I'll report back when it's done.
Still very slow in CI even with that setting. It's been just over an hour and it's still running. Seeing lots of these:
Copy code
00:16:43.15 [INFO] Long running tasks:
  60.80s	Building pytest_runner.pex
00:17:13.19 [INFO] Long running tasks:
  90.83s	Building pytest_runner.pex
00:17:43.24 [INFO] Long running tasks:
  120.89s	Building pytest_runner.pex
00:18:13.31 [INFO] Long running tasks:
  150.95s	Building pytest_runner.pex
00:18:43.41 [INFO] Long running tasks:
  181.05s	Building pytest_runner.pex
00:19:13.46 [INFO] Long running tasks:
  211.10s	Building pytest_runner.pex
00:19:43.50 [INFO] Long running tasks:
  241.14s	Building pytest_runner.pex
h
So how that environment is different from your local one? Probably time to dig in on that. See if that machine is swapping aggressively, and so on
n
Yeah - that's rather hard to do because it's a GitLab shared runner. But my CI jobs used to run fine - in about 15 minutes and then starting yesterday they've slowed to a crawl - I haven't been able to get 1 to finish with a 2 hour timeout. And it's not even the step that running any of my code that takes time - it's all the 
.pex
file building. For example:
Copy code
22:19:46.05 [INFO] Starting: Building mypy.pex from <resource://pants.backend.python.typecheck.mypy/mypy.lock>
22:19:46.17 [INFO] Starting: Installing 3rdparty/python/default.lock for the resolve `default`
22:19:46.26 [INFO] Canceled: Building mypy.pex from <resource://pants.backend.python.typecheck.mypy/mypy.lock>
22:19:46.26 [INFO] Starting: Building mypy.pex from <resource://pants.backend.python.typecheck.mypy/mypy.lock>
22:19:54.03 [INFO] Completed: Building mypy.pex from <resource://pants.backend.python.typecheck.mypy/mypy.lock>
22:19:54.03 [INFO] Completed: Scheduling: Building mypy.pex from <resource://pants.backend.python.typecheck.mypy/mypy.lock>
22:21:11.29 [INFO] Long running tasks:
  85.12s	Installing 3rdparty/python/default.lock for the resolve `default`
22:21:41.37 [INFO] Long running tasks:
  115.20s	Installing 3rdparty/python/default.lock for the resolve `default`
22:22:11.41 [INFO] Long running tasks:
  145.24s	Installing 3rdparty/python/default.lock for the resolve `default`
22:22:30.19 [INFO] Completed: Installing 3rdparty/python/default.lock for the resolve `default`
22:22:30.19 [INFO] Completed: Scheduling: Installing 3rdparty/python/default.lock for the resolve `default`
22:22:30.20 [INFO] Starting: Building requirements.pex from 3rdparty/python/default.lock
22:23:41.59 [INFO] Long running tasks:
  71.39s	Building requirements.pex from 3rdparty/python/default.lock
22:24:00.18 [INFO] Completed: Building requirements.pex from 3rdparty/python/default.lock
22:24:00.18 [INFO] Completed: Scheduling: Building requirements.pex from 3rdparty/python/default.lock
22:24:00.21 [INFO] Starting: Building requirements_venv.pex
22:25:11.78 [INFO] Long running tasks:
  71.57s	Building requirements_venv.pex
22:25:41.85 [INFO] Long running tasks:
  101.64s	Building requirements_venv.pex
22:26:11.92 [INFO] Long running tasks:
  131.71s	Building requirements_venv.pex
22:26:42.00 [INFO] Long running tasks:
  161.79s	Building requirements_venv.pex
22:26:48.50 [INFO] Completed: Building requirements_venv.pex
22:26:48.57 [INFO] Completed: Scheduling: Building requirements_venv.pex
22:26:52.27 [INFO] Completed: Scheduling: Determine distributions found in mypy.pex
Note also that it 
Canceled: Building mypy.pex
then started it again. Is that a clue?
AH! I figured it out!!! Debian trixie change 
/tmp
so it's an in-memory tempfs which means the default pants setup, which uses 
/tmp
for sandboxes quickly fills up RAM and the job dies. So I added:
Copy code
local_execution_root_dir = "~/.cache/pants-sandboxes"
to my 
pants.toml
. Apparently, on the CI servers that maps to some crazy slow storage somehow and everything slows to a crawl. it's odd because th CI runs in a docker container so I'd expect 
/tmp
and 
~/.cache
to both be inside the same container overlayed over the same underlying storage but somehow that makes all the difference in the world.
64 Views
Open in Slack
PreviousNext
Powered by