Hello everyone, I recently introduced pants remote caching (via

bazel-remote

) to our CI usage and for some jobs I'm noticing the linters/checkers actually taking more time even though the requirements are cached. I'm gonna split it into two separate problems I'm seeing. First is that

mypy

process itself seems to take double the time when requirements were cached. Original run logs with requirement list info messages truncated for clarity:

11:29:25.49 [INFO] Starting: Building 14 requirements for mypy.pex
11:29:25.49 [INFO] Starting: Building 174 requirements for requirements.pex
11:29:25.62 [INFO] Canceled: Building 14 requirements for mypy.pex
11:29:25.62 [INFO] Starting: Building 14 requirements for mypy.pex
11:29:37.65 [INFO] Completed: Building 14 requirements for mypy.pex
11:30:33.18 [INFO] Long running tasks:
  67.69s        Building 174 requirements for requirements.pex
11:30:48.63 [INFO] Completed: Building 174 requirements for requirements.pex
11:30:48.71 [INFO] Starting: Building requirements_venv.pex
11:32:03.24 [INFO] Long running tasks:
  74.53s        Building requirements_venv.pex
11:32:33.25 [INFO] Long running tasks:
  104.54s       Building requirements_venv.pex
11:33:03.28 [INFO] Long running tasks:
  134.58s       Building requirements_venv.pex
11:33:27.75 [INFO] Completed: Building requirements_venv.pex
11:34:33.42 [INFO] Long running tasks:
  64.57s        Run MyPy on 2630 files.
11:35:03.47 [INFO] Long running tasks:
  94.63s        Run MyPy on 2630 files.
11:35:33.53 [INFO] Long running tasks:
  124.68s       Run MyPy on 2630 files.
11:36:03.61 [INFO] Long running tasks:
  154.76s       Run MyPy on 2630 files.
11:36:10.87 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.

Requirement building takes about 4m, mypy itself takes about 3 minutes With caching:

00:16:30.67 [INFO] Starting: Building 14 requirements for mypy.pex
00:16:30.86 [INFO] Canceled: Building 14 requirements for mypy.pex
00:16:31.13 [INFO] Starting: Building 174 requirements for requirements.pex
00:16:31.34 [INFO] Canceled: Building 174 requirements for requirements.pex
00:16:31.42 [INFO] Starting: Building requirements_venv.pex
00:16:31.64 [INFO] Canceled: Building requirements_venv.pex
00:17:56.80 [INFO] Long running tasks:
  84.82s        Run MyPy on 2630 files.
00:18:26.89 [INFO] Long running tasks:
  114.91s       Run MyPy on 2630 files.
00:18:56.99 [INFO] Long running tasks:
  145.00s       Run MyPy on 2630 files.
00:19:27.03 [INFO] Long running tasks:
  175.04s       Run MyPy on 2630 files.
00:19:57.05 [INFO] Long running tasks:
  205.06s       Run MyPy on 2630 files.
00:20:27.09 [INFO] Long running tasks:
  235.10s       Run MyPy on 2630 files.
00:20:57.10 [INFO] Long running tasks:
  265.12s       Run MyPy on 2630 files.
00:21:27.14 [INFO] Long running tasks:
  295.15s       Run MyPy on 2630 files.
00:21:51.72 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.

Requirement building is seconds because it's retrieved from cache, but now mypy takes 5.5minutes? Looking at resource graphs memory/CPU looks about the same. What could be causing the mypy process itself to take longer?