Should `yapf` use ` parallel` if ` fmt per file caching` is Pants #development

Should `yapf` use `--parallel` if `--fmt-per-file-...

bitter-ability-32190

12/23/2021, 5:59 PM

Should

yapf

use

--parallel

--fmt-per-file-caching

is off? 🧵

bitter-ability-32190

12/23/2021, 5:59 PM

The difference is about 2 minutes without, and maybe 20 seconds with for my ~2000 files?

hundreds-father-404

12/23/2021, 6:02 PM

Could you please comment on https://github.com/pantsbuild/pants/issues/9964 mentioning that? It's a bit nuanced because we don't want contention. If you're running 6 other linters in parallalel, and they themselves are concurrent, you get too much concurrency

bitter-ability-32190

12/23/2021, 6:03 PM

Note: specifcally I'm looking at

fmt

😉

👍 1

hundreds-father-404

12/23/2021, 6:04 PM

That could make sense to have it enabled for

fmt

but not

lint

. Although still slightly tricky because Pants will run formatters in parallel if they're from different languages. Gofmt, Shfmt, Prettier, Cargo fmt, and Yapf all might be running at the same time. It's only sequential within each language

☝️ 1

bitter-ability-32190

12/23/2021, 6:07 PM

I don't think there's an issue for it but I think this belongs in the "allow batched runs" feature mentioned here: https://github.com/pantsbuild/pants/issues/13462

bitter-ability-32190

12/23/2021, 6:09 PM

Unfortunately I have no idea how it'd relate to remote execution, but ideally you could batch requests, and maybe even in the same # of processes (to take advantage of tools' in-memory caches). Of course that involves have parseable output. It's probably worth floating the idea to various tools to have some kind of machine-parsable output 🤷‍♂️

2 Views

Open in Slack

Previous Next