#23180 Towards Public Benchmarks New discussion created by cburroughs Fast is good! People are endeared to fast tools. Faster is a feature and makes new things possible. Fast makes it feasible to compose commands into more useful workflows. Historically most reasoning about performance for the project has centered around either the Pants repository itself, or private ones. These aren't bad approaches, but each of these has downsides. Pants itself has meaningful differences from Pants users such as: • Is maximal in backend use. Most users don't use every language or backend. • Has few "deployable" targets. • Is narrowly constrained in 3rdparty dependencies (no giant Python ML libraries) • Is as large as it needs to be; not among the biggest or smallest repositories. • Always bootstraps at HEAD. Can't (easily) use Pants with older Pants versions to compare performance. Relying on internal repositories has the great virtue of measuring real world use, but the obvious downside that they can not be shared. There is also always uncertainty in judging how representative an internal process is. Is Pants slow at X? Or is X something absurd where the answer for any tool would be "don't do that"? In connection with #18911 @jriddy did some great work to try to create a synthetic repository that preserved the structure of an internal repository, but with all of the symbols renamed and bodies of functions removed. This wasn't perfect -- the original pathological behavior wasn't replicated -- but it was still super useful and I think this is a promising model. So: a benchmark derived from real use that is not totally synthetic or artificial. Ultra processed benchmarks perhaps? As an aspiration, I'd imagine something like The Python Performance Benchmark Suite where there are a set of benchmarks and a tool to help consistently run them and aggregate results. Like all parts of a community project, which benchmarks exist is partly a function of who is motivated to make one and what they are interested in. I think https://github.com/cburroughs/clam-diggers/ is great for Python dependency inference. Some other areas I might be interested in are: visibility,

check

hot cache path, changed-since. Some Open & Discussion Questions • How to structure the relationship between the harness and the benchmarks? Inline? Sub-modules? Have a checkout script? A lot of tradeoffs here. • I'm assuming on a scale between "macro" and "micro" benchmarks, that everyone is going to say "macro" is better. But

pyperformance

is a somewhat idiosyncratic mix. I can think of a few cases where something bugs me for how slow it is, but to get it to clearly stand out I'd have to make a pretty odd repository. • Does anyone care about absolute/consistent numbers, or is it always about change by change relative performance? "How was your Python compiled?" makes a huge difference for example, and one could sink a lot of time in environment consistency. • I suspect it is going to be easier to get consistent results for CPU/memory heavy problems vs IO. But "doing anything with this giant pile of dependencies is slow" is a common -- and very reasonable -- compliant. • I'm super excited about the embarrassment of riches for improvements from the `perf` support. I don't have a good sense of how valuable "Run the benchmark in docker with everything properly compiled" is. Or is there some equivalent mac thing people have been using? • How could open source Pants use like https://github.com/StackStorm/st2/ fit into this? pantsbuild/pants