https://pantsbuild.org/ logo
Powered by
e

enough-analyst-54434

12/05/2022, 6:11 PM
Experiments with a new pants launcher have revealed:
Copy code
$ hyperfine -w5 'PANTS_VERSION=2.12.0 scie-pants -V' 'PANTS_VERSION=2.13.0 scie-pants -V' 'PANTS_VERSION=2.14.0 scie-pants -V'
Benchmark 1: PANTS_VERSION=2.12.0 scie-pants -V
  Time (mean ± σ):     390.1 ms ±  10.8 ms    [User: 308.7 ms, System: 19.8 ms]
  Range (min … max):   375.8 ms … 405.7 ms    10 runs

Benchmark 2: PANTS_VERSION=2.13.0 scie-pants -V
  Time (mean ± σ):     506.4 ms ±  17.8 ms    [User: 396.4 ms, System: 18.9 ms]
  Range (min … max):   482.3 ms … 543.2 ms    10 runs

Benchmark 3: PANTS_VERSION=2.14.0 scie-pants -V
  Time (mean ± σ):     525.6 ms ±  12.9 ms    [User: 414.0 ms, System: 24.8 ms]
  Range (min … max):   514.4 ms … 546.7 ms    10 runs

Summary
  'PANTS_VERSION=2.12.0 scie-pants -V' ran
    1.30 ± 0.06 times faster than 'PANTS_VERSION=2.13.0 scie-pants -V'
    1.35 ± 0.05 times faster than 'PANTS_VERSION=2.14.0 scie-pants -V'
Old news or was the perf regression from 2.12 - 2.13 well known?
👀 2
I mean, its only ~100ms and Pants is ~10x slower than that with the existing ./pants launcher script mixed in; so maybe we're just a boiling frog here.
h

happy-kitchen-89482

12/05/2022, 6:15 PM
Hmmm, not known to me at least, and somewhat concerning. Is this with the native client?
e

enough-analyst-54434

12/05/2022, 6:15 PM
No
The scie-pants is ~33% faster for existing pantsbuild.pants distros on noop runs.
But thats washed out here since its all 1 hyperfine run using the same pants launcher. The relative difference is the thing.
And its all Python 3.9.15.
So, about as fast as it can get presumably.
Ok, I think question answered. This just goes towards my long embarrassment of not having automated historical perf-tracking as part of CI for Pants or Pex. Clearly not embarrassed enough yet, but I need to get there and do something about this.
I've been using https://github.com/BruceEckel/tmp-pants-bug as my test repo FWIW since I had been doing some support there and it was handy.
c

curved-television-6568

12/05/2022, 6:25 PM
e

enough-analyst-54434

12/05/2022, 6:26 PM
Towards this end, and https://pantsbuild.slack.com/archives/G0107QFGGAW/p1667850768538119?thread_ts=1667850597.632339&cid=G0107QFGGAW I'm going to create a pantsbuild/scie-pants repo to move my prototype to for review. We can always kill the repo if the prototype is deemed inappropriate in the end, but I'd like to start getting the code out for review instead of in my private scie-pants repo.
✅ 2
@curved-television-6568 my tests use -w5 to warmup run 5x before each benchmark exactly to avoid scheduler initialization. So my results are likely unrelated.
If I 
--show-output
IOW there are 0 scheduler initializations in the benchmark runs.
c

curved-television-6568

12/05/2022, 6:28 PM
Ah, gotcha 👍
w

wide-midnight-78598

12/05/2022, 6:35 PM
Is this equivalent to running the 
./pants
bash script, or does it bypass that?
e

enough-analyst-54434

12/05/2022, 7:02 PM
Its the equivalent ... and it bypasses that! Its pure rust in the warm case (<1ms to re-exec into Pants venv 
pants
console script) and an installer.pex for the cold case when a Pants version has never been installed (supports PANTS_SHA, PANTS_DEBUG, etc), but faster and can be anywhere on $PATH - no need to check into repo.
Ok, end of the "fun", the historical record with 1 statistical outlier in this run for 2.5.0. N.B.: All runs prior to 2.5.0 use Python 3.8.15, all after 3.9.15:
Copy code
$ hyperfine -w2 -L version 1.30.5rc1,2.0.0,2.1.0,2.2.0,2.3.0,2.4.0,2.5.0,2.6.0,2.7.0,2.8.0,2.9.0,2.10.0,2.11.0,2.12.0,2.13.0,2.14.0,2.15.0a1,2.16.0.d
ev0 'PANTS_VERSION={version} PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
Benchmark 1: PANTS_VERSION=1.30.5rc1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     626.2 ms ±  32.3 ms    [User: 393.4 ms, System: 177.6 ms]
  Range (min … max):   587.5 ms … 679.5 ms    10 runs

Benchmark 2: PANTS_VERSION=2.0.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     399.6 ms ±  16.5 ms    [User: 298.0 ms, System: 30.0 ms]
  Range (min … max):   380.8 ms … 431.9 ms    10 runs

Benchmark 3: PANTS_VERSION=2.1.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     319.4 ms ±   6.8 ms    [User: 250.3 ms, System: 22.2 ms]
  Range (min … max):   311.0 ms … 331.2 ms    10 runs

Benchmark 4: PANTS_VERSION=2.2.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     312.8 ms ±  14.4 ms    [User: 243.9 ms, System: 23.2 ms]
  Range (min … max):   300.1 ms … 348.0 ms    10 runs

Benchmark 5: PANTS_VERSION=2.3.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     310.0 ms ±   3.7 ms    [User: 239.4 ms, System: 20.5 ms]
  Range (min … max):   307.0 ms … 319.8 ms    10 runs

Benchmark 6: PANTS_VERSION=2.4.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     332.0 ms ±   8.5 ms    [User: 236.7 ms, System: 34.1 ms]
  Range (min … max):   322.2 ms … 351.0 ms    10 runs

Benchmark 7: PANTS_VERSION=2.5.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     421.0 ms ± 290.6 ms    [User: 251.0 ms, System: 18.9 ms]
  Range (min … max):   324.1 ms … 1247.9 ms    10 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 8: PANTS_VERSION=2.6.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     322.8 ms ±   4.3 ms    [User: 247.4 ms, System: 16.0 ms]
  Range (min … max):   317.1 ms … 331.9 ms    10 runs

Benchmark 9: PANTS_VERSION=2.7.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     338.7 ms ±   9.9 ms    [User: 253.1 ms, System: 22.7 ms]
  Range (min … max):   328.7 ms … 354.5 ms    10 runs

Benchmark 10: PANTS_VERSION=2.8.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     337.3 ms ±   7.8 ms    [User: 261.0 ms, System: 17.0 ms]
  Range (min … max):   328.8 ms … 357.6 ms    10 runs

Benchmark 11: PANTS_VERSION=2.9.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     344.4 ms ±   7.3 ms    [User: 273.5 ms, System: 8.0 ms]
  Range (min … max):   336.2 ms … 358.2 ms    10 runs

Benchmark 12: PANTS_VERSION=2.10.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     351.4 ms ±   8.0 ms    [User: 291.7 ms, System: 12.2 ms]
  Range (min … max):   342.6 ms … 369.2 ms    10 runs

Benchmark 13: PANTS_VERSION=2.11.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     372.4 ms ±   5.8 ms    [User: 297.6 ms, System: 23.3 ms]
  Range (min … max):   363.7 ms … 383.5 ms    10 runs

Benchmark 14: PANTS_VERSION=2.12.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     369.1 ms ±   8.7 ms    [User: 291.7 ms, System: 27.6 ms]
  Range (min … max):   358.9 ms … 388.7 ms    10 runs

Benchmark 15: PANTS_VERSION=2.13.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     460.8 ms ±   7.7 ms    [User: 373.4 ms, System: 16.9 ms]
  Range (min … max):   452.7 ms … 480.5 ms    10 runs

Benchmark 16: PANTS_VERSION=2.14.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     493.3 ms ±  10.8 ms    [User: 406.6 ms, System: 19.7 ms]
  Range (min … max):   481.9 ms … 516.8 ms    10 runs

Benchmark 17: PANTS_VERSION=2.15.0a1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     491.1 ms ±  11.8 ms    [User: 383.3 ms, System: 24.0 ms]
  Range (min … max):   477.6 ms … 514.4 ms    10 runs

Benchmark 18: PANTS_VERSION=2.16.0.dev0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V
  Time (mean ± σ):     493.8 ms ±  11.4 ms    [User: 392.2 ms, System: 17.9 ms]
  Range (min … max):   483.0 ms … 515.9 ms    10 runs

Summary
  'PANTS_VERSION=2.3.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V' ran
    1.01 ± 0.05 times faster than 'PANTS_VERSION=2.2.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.03 ± 0.03 times faster than 'PANTS_VERSION=2.1.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.04 ± 0.02 times faster than 'PANTS_VERSION=2.6.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.07 ± 0.03 times faster than 'PANTS_VERSION=2.4.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.09 ± 0.03 times faster than 'PANTS_VERSION=2.8.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.09 ± 0.03 times faster than 'PANTS_VERSION=2.7.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.11 ± 0.03 times faster than 'PANTS_VERSION=2.9.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.13 ± 0.03 times faster than 'PANTS_VERSION=2.10.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.19 ± 0.03 times faster than 'PANTS_VERSION=2.12.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.20 ± 0.02 times faster than 'PANTS_VERSION=2.11.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.29 ± 0.06 times faster than 'PANTS_VERSION=2.0.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.36 ± 0.94 times faster than 'PANTS_VERSION=2.5.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.49 ± 0.03 times faster than 'PANTS_VERSION=2.13.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.58 ± 0.04 times faster than 'PANTS_VERSION=2.15.0a1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.59 ± 0.04 times faster than 'PANTS_VERSION=2.14.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    1.59 ± 0.04 times faster than 'PANTS_VERSION=2.16.0.dev0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
    2.02 ± 0.11 times faster than 'PANTS_VERSION=1.30.5rc1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants -V'
Summary: Yay Pants 2!, but then the slow boiling begins again.
👍 1
h

happy-kitchen-89482

12/05/2022, 7:16 PM
Hypothesis with as yet no evidence, so treat skeptically: this may be due to option registration, and the startup time correlates with the total number of options. The reason I advance this hypothesis is that is that back in the v1 days I profiled the startup time and it was dominated by option registration.
Or, maybe that jump correlates with the switch to declarative options?
🙊 1
e

enough-analyst-54434

12/05/2022, 7:19 PM
Well, the backends here is fixed to Bruce's example list of python + 4 lint backends (minus mypy which was either not a backend or had a different name in 1.30 so I had to edit it out). I forget, is it the case that all options are registered even when a backend is not selected?
h

happy-kitchen-89482

12/05/2022, 7:20 PM
No, that shouldn’t be the case
But we do add subsystems in existing backends, and options in existing subsystems, over time
e

enough-analyst-54434

12/05/2022, 7:21 PM
Ok, well, to m continued embarrassment again, I'll drop this here having ~pinpointed a step in the otherwise slow temperature increase.
h

happy-kitchen-89482

12/05/2022, 7:22 PM
The declarative options were rolled out in 2.11, so that doesn’t account for the jump
I’ve been trying to count the total number of options in each version, but that exposed a bug: https://github.com/pantsbuild/pants/issues/17715
So it’s an ill wind that blows nobody any good
After fixing that, time for one of my favorite activities, CPU profiling!
e

enough-analyst-54434

12/05/2022, 7:39 PM
I lied, 1 more for @curved-television-6568. You were right I think. I huge 500ms bump for scheduler initialization (
--no-pantsd
is the test this time) from 2.14.0 to 2.15.0a2. Note here the slow boiling seems to be more obvious than the 
--pantsd
default case, but this step in 2.15.x is an even bigger step than the 
--pantsd
case in the 2.12.0 to 2.13.0 transition:
Copy code
$ hyperfine -w2 -L version 1.30.5rc1,2.0.0,2.1.0,2.2.0,2.3.0,2.4.0,2.5.0,2.6.0,2.7.0,2.8.0,2.9.0,2.10.0,2.11.0,2.12.0,2.13.0,2.14.0,2.15.0a1,2.16.0.dev0 'PANTS_VERSION={version} PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
Benchmark 1: PANTS_VERSION=1.30.5rc1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     907.4 ms ±  59.0 ms    [User: 634.1 ms, System: 185.8 ms]
  Range (min … max):   835.8 ms … 1041.5 ms    10 runs

Benchmark 2: PANTS_VERSION=2.0.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     750.8 ms ±  32.1 ms    [User: 630.0 ms, System: 42.7 ms]
  Range (min … max):   724.3 ms … 834.2 ms    10 runs

Benchmark 3: PANTS_VERSION=2.1.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     683.4 ms ±  14.7 ms    [User: 571.0 ms, System: 35.4 ms]
  Range (min … max):   653.0 ms … 697.9 ms    10 runs

Benchmark 4: PANTS_VERSION=2.2.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     694.4 ms ±  19.9 ms    [User: 575.0 ms, System: 33.4 ms]
  Range (min … max):   668.0 ms … 733.4 ms    10 runs

Benchmark 5: PANTS_VERSION=2.3.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     854.1 ms ±  23.1 ms    [User: 678.4 ms, System: 41.4 ms]
  Range (min … max):   826.0 ms … 903.9 ms    10 runs

Benchmark 6: PANTS_VERSION=2.4.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     933.4 ms ±  15.3 ms    [User: 765.6 ms, System: 42.6 ms]
  Range (min … max):   917.0 ms … 955.8 ms    10 runs

Benchmark 7: PANTS_VERSION=2.5.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     984.4 ms ±  17.6 ms    [User: 801.2 ms, System: 55.4 ms]
  Range (min … max):   950.9 ms … 1012.9 ms    10 runs

Benchmark 8: PANTS_VERSION=2.6.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     947.4 ms ±  25.2 ms    [User: 762.6 ms, System: 54.5 ms]
  Range (min … max):   918.6 ms … 989.3 ms    10 runs

Benchmark 9: PANTS_VERSION=2.7.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.013 s ±  0.017 s    [User: 0.836 s, System: 0.049 s]
  Range (min … max):    0.998 s …  1.044 s    10 runs

Benchmark 10: PANTS_VERSION=2.8.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):     936.2 ms ±  25.5 ms    [User: 771.0 ms, System: 42.5 ms]
  Range (min … max):   900.7 ms … 990.9 ms    10 runs

Benchmark 11: PANTS_VERSION=2.9.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.009 s ±  0.018 s    [User: 0.830 s, System: 0.050 s]
  Range (min … max):    0.979 s …  1.031 s    10 runs

Benchmark 12: PANTS_VERSION=2.10.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.018 s ±  0.031 s    [User: 0.846 s, System: 0.043 s]
  Range (min … max):    0.979 s …  1.066 s    10 runs

Benchmark 13: PANTS_VERSION=2.11.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.064 s ±  0.024 s    [User: 0.895 s, System: 0.047 s]
  Range (min … max):    1.025 s …  1.097 s    10 runs

Benchmark 14: PANTS_VERSION=2.12.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.122 s ±  0.016 s    [User: 0.945 s, System: 0.053 s]
  Range (min … max):    1.084 s …  1.139 s    10 runs

Benchmark 15: PANTS_VERSION=2.13.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.188 s ±  0.014 s    [User: 1.019 s, System: 0.048 s]
  Range (min … max):    1.170 s …  1.213 s    10 runs

Benchmark 16: PANTS_VERSION=2.14.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.242 s ±  0.022 s    [User: 1.078 s, System: 0.048 s]
  Range (min … max):    1.202 s …  1.284 s    10 runs

Benchmark 17: PANTS_VERSION=2.15.0a1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.761 s ±  0.026 s    [User: 1.571 s, System: 0.064 s]
  Range (min … max):    1.724 s …  1.816 s    10 runs

Benchmark 18: PANTS_VERSION=2.16.0.dev0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V
  Time (mean ± σ):      1.781 s ±  0.025 s    [User: 1.573 s, System: 0.074 s]
  Range (min … max):    1.755 s …  1.830 s    10 runs

Summary
  'PANTS_VERSION=2.1.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V' ran
    1.02 ± 0.04 times faster than 'PANTS_VERSION=2.2.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.10 ± 0.05 times faster than 'PANTS_VERSION=2.0.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.25 ± 0.04 times faster than 'PANTS_VERSION=2.3.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.33 ± 0.09 times faster than 'PANTS_VERSION=1.30.5rc1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.37 ± 0.04 times faster than 'PANTS_VERSION=2.4.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.37 ± 0.05 times faster than 'PANTS_VERSION=2.8.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.39 ± 0.05 times faster than 'PANTS_VERSION=2.6.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.44 ± 0.04 times faster than 'PANTS_VERSION=2.5.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.48 ± 0.04 times faster than 'PANTS_VERSION=2.9.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.48 ± 0.04 times faster than 'PANTS_VERSION=2.7.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.49 ± 0.06 times faster than 'PANTS_VERSION=2.10.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.56 ± 0.05 times faster than 'PANTS_VERSION=2.11.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.64 ± 0.04 times faster than 'PANTS_VERSION=2.12.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.74 ± 0.04 times faster than 'PANTS_VERSION=2.13.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    1.82 ± 0.05 times faster than 'PANTS_VERSION=2.14.0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    2.58 ± 0.07 times faster than 'PANTS_VERSION=2.15.0a1 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
    2.61 ± 0.07 times faster than 'PANTS_VERSION=2.16.0.dev0 PANTS_VERIFY_CONFIG=False PANTS_BACKENDS=-["pants.backend.python.typecheck.mypy"] scie-pants --no-pantsd -V'
Aha, Stu knew that. Environments work. So the 2.14 -> 2.15 scheduler initialization time increase is known / shot called / accepted-ish for now, and it's just the 2.12 -> 2.13 hot path ~100ms increase that is unknown / un-accounted for.
👍 1
h

happy-kitchen-89482

12/05/2022, 11:14 PM
PS The actual option-counting bug was in my brain, not in the code…
👌 1
There was no substantial change in the number of options in those backends between 2.12.0 and 2.13.0
So that hypothesis is unlikely, unless something made options parsing significantly slower in 2.13.0. Profiling will reveal all.
Powered by