I've got a decent sized mono-repo with Python code in it building with Pants. Some operations are very slow. In particular

pants generate-lockfiles

and

pants check

are very slow if any dependencies have changed. For

generate-lockfiles

we get lots of messages like:

17:48:30.91 [INFO] Long running tasks:
  60.07s        Generate lockfile for python-default

And those runs can take upwards of 10 minutes. For

check

the slowness is building a set of dependencies for a

mypy

run for each target to be type checked. I think the slowness is largely because we use PyTorch (things slowed WAAAY down when we added our first PyTorch dependency) and I think I read that to properly resolve versions sometimes whole wheels have to be downloaded which means downloading multiple giant PyTorch wheels. In any event, regardless of the cause, I'm wondering if there's a way to speed these things up. In particular I know you can use different resolves. If we put all our expensive-to-resolve things like PyTorch into one resolve and everything else into another one I think we can use dependencies from each resolve in the same code (right?) and I'm hoping that then modifying dependencies for the non-PyTorch resolve would be fast again - and most of our dependency changes aren't PyTorch. Questions: 1. Would the multiple-resolve things work? 2. How do you set it up so a single library or executable can use dependencies from both resolves easily? 3. Is there a better way to speed this kind of thing up?

Are you on Pants 2.21?

Nope. I'm on 2.20.1

Yeah, specifically some parts involving pulling down PyTorch. That’s not some panacea for everything, just pointing it out.

specifically some parts involving pulling down PyTorch

So in 2.20 in order to find the right version of PyTorch it pulls multiple wheels but in 2.21 it knows how to do the dependency resolution without doing that?? That sounds great!

👍 Please report back if it sped anything up, and by how much - would be good to have more data points. I’ve tested it on some repos with smaller deps, and seeing something closer to 10-15% perf improvements is closer to the norm (but PyTorch is its own world already)

So in 2.20 in order to find the right version of PyTorch it pulls multiple wheels but in 2.21 it knows how to do the dependency resolution without doing that?? That sounds great!

I think dependency resolution (i.e.

generate-lockfiles

) hasn't changed, but... previously for anything actually using the dependencies like

pants check

, it was previously unzipping and re-pingzip the PyTorch wheels, but can now just use the raw wheels as download.

It'd be nice if

generate-lockfiles

could be faster but

pants check

was indeed very slow and it sounds like this will help. Building pex files was also quite slow - do you know if it'll help with that as well? I'm in the middle of a large refactor but when done I'll try to bump the pants version and let you know how it goes. Thanks.

Would you say it’s slower on 2.21 or faster? Just trying to get a read on whether it was an improvement or not - in isolation, and to the workflow overall (e.g. speeding up one part, but slowing down another might be a net negative)

I think creating lockfiles is unchanged. I think building pex files is quite a bit faster. tests and mypy runs after a dependency change seem like they're also unchanged. I should have done some benchmarks before the upgrade but didn't so this is all just subjective.

Re generate-lockfiles speed pip 24 (not yet in pants; pr in progress) has some relevant performance improvements, with more in trunk for the next pip release. I'm hopeful it helps somewhat on the margins.