Why can't

lint-per-file-process

use

repository.pex

directly?

You can use this option in 2.9 to skip that: https://www.pantsbuild.org/v2.9/docs/reference-python#section-run-against-entire-lockfile See "Fine-grained resolves without a performance hit" in https://blog.pantsbuild.org/introducing-pants-2-5/ for what's going on

Oh interesting. in the short term that's certainly faster. I wonder if the long-term which one will win 🤔

I think the other way around - with a cold cache, it's more work because you have to do the extraction step. But it means a warm cache has much less invalidation

I meant in terms of weeks. So long as our lockfile never changes thats faster. But when it changes we pay the penalty. Of course if the work to resolve all the little requirements.pex is more than the work it takes to rebuild the big kahuna, it's still worth it

I wonder if repository.pex can be (and is) remotely cached between 2 disperate systems

Remote caching works awesome currently in CI, where you have multiple distinct machines but all with the same PATH and setup etc It doesn't work as well with desktop usage because systems tend to diverge in ways that invalidate the cache. https://github.com/pantsbuild/pants/issues/12203 tracks how we might improve that

note also though that there are some significant improvements to subsetting in

2.9.0rc2

: see https://github.com/pantsbuild/requirements-perf/issues/2

We should highlight that in release blog

and there is another bit of low hanging fruit with subsetting: we’ve added support for reusing immutable inputs to sandboxes, and the

repository.pex

should definitely be one. https://github.com/pantsbuild/pants/issues/14070

Should the path be an input to creating the repository pex?

currently: yes unfortunately. many python artifact builds require e.g. a C compiler

Ah right

addressing https://github.com/pantsbuild/pants/issues/12203 without boiling the entire ocean will be a fine line to walk

I think the measurement based approach is a good one

i’m guessing that pylint has a fair amount of per-file overhead, as described in the doc that Eric linked

Yes

that’s maybe not surprising… using

run-against-entire-lockfile

means materializing all of your requirements for each lint process

Ahhhhh that might be it

and then if

pylint

is actually consuming all of them…?

Which we can fix via https://github.com/pantsbuild/pants/issues/14070 right?

@hundreds-father-404 I was just thinking that 🙂

@bitter-ability-32190: to confirm, you could use your flamegraphy tool to look at sandbox creation time: that workunit is called

setup_sandbox

I could, yes. I'd just need to dust it off and context switch 😂

but time spent in pylint will not be.

For the curious, I assume that involves parsing the output of the tools. Any ideas on how to do that?

i’m hoping to constrain what a

@rule

is able to consume when the feature is enabled… we’ll see how feasible that is.

@witty-crayon-22786 here’s a live example -

@melodic-thailand-99227: the

--pex-verbosity

setting (https://www.pantsbuild.org/docs/reference-pex#section-verbosity) can render what PEX is actually doing there… could try with

3

or so, and see if it highlights how time is being spent. but as mentioned, it should all just be time spent in pip.

(Commenting on the issue as well for vis) So it turns out there is a "hack" that can be used in the meantime. Setting the relevant parallelism options for each tool turns out to be a "poor-mans-batch" is faster than not setting it (obviously) but also faster than the

-per-file-caching

. E.g. If I wanted to run

fmt

with

isort

and

black

and I have a 64 core machine. Running with

--black-args='-W 64' --isort-args='--jobs=64'

is always the fastest option. Whereas with

lint

(which runs in parallel) you probably want to take N cores and divide by M checkers...

--black-args='-W 16' --isort-args='--jobs=16' --flake8-args='--jobs=16' --pylint-args='--jobs=16'

. Depending on how long fixing the issue might take, it might be worth finding/writing clever little plugins to set the num jobs smartly based on CPU count

that is really useful data: thanks! https://github.com/pantsbuild/pants/issues/9964 suggests that we might be able calculate that value dynamically

Edit: I misunderstood the issue linked

thanks.

radical