I wanted to compare the performance of Pants dependency inference to perhaps some naive ways of doing it myself, and I'm getting quite better performance with a single-threaded AST scanning method. I know this method doesn't have anywhere near the number of features as Pants does, but I want to better understand why my silly script (12-15 sec) performs so much better than Pants (30-45 sec) for the same codebase and queries? I assume its something involving cache, IO, or task scheduler overhead, but I'd like to hear y'all's input. More details in 🧵

Pants does just use

ast

to parse the inputs. The Pants rule code is here: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/dependency_inference/parse_python_imports.py and the parser is here: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/dependency_inference/scripts/import_parser.py

keep in mind that pants also looks at strings literals that not just import statements (this is useful with Django for example, where strings are used in setting files to reference various components)

keep in mind that pants also looks at strings literals

Only if

[python-infer].strings

is true

to measure pants perf, I'm nuking the lmdb_store and running

./pants dependees ...

and timing that

Ah re

dependees

, I strongly suspect https://github.com/pantsbuild/pants/issues/11270 is in play

Looking at this code, the big difference is that Pants is using a

Get

for every little thing, I would suspect there's some resource contention at the scheduler level

I've thought about batching dependency inference, but never surfaced the thought here

I assume its something involving cache, IO, or task scheduler overhead,

One source of overhead is determining the Python interpreter to parse the code with, which can be 0.25-2 seconds I believe. That does get memoized w/ pantsd, but not helpful on cold run. Instead, your script uses the interpreter its executed with

if it were 1-2 sec difference i wouldn't worry about it

Just run Pants and your script 3 times, then they're even (cuz' caching)

Josh there is also overhead in launching a distinct process per file to parse imports, vs doing in-memory. But the benefit is caching to disk. And we need to do it that way because interpreter selection We could batch that more, but then we lose fine-grained caching. Any change at all to the file, even whitespace, invalidates the cache, so I think fine-grained cache is pretty crucial. Altho could be worth experimenting w/ batching

I think the caching makes sense

things like https://github.com/pantsbuild/pants/issues/13112 should hopefully make a material difference here. we’re definitely working to push down our overhead.

I think similar to lint or fmt, if e batched smartly it might be a win. It should be easy to come up with numbers

Yeah, I'd guess the big difference here is the overhead of launching a process per file to parse

Yeah that'd benefit others like fmt and lint as well