did i miss any outstanding performance issues? <ht...
# development
w
did i miss any outstanding performance issues? https://github.com/orgs/pantsbuild/projects/2/views/1
(the PEX_PATH transitivity ticket is a placeholder for a new issue covering the issue of improving subsetting)
b
That page 404s for ne
w
ah, interesting… apparently beta Projects have visibility. fixed!
h
Rust native client
w
…ah, yea. we don’t have a ticket for the remaining work there. thanks!
m
pants resolver takes a lot of time even if we give him hashes of all files - 10-15 minutes is not unusual
w
@melodic-thailand-99227: that’s pretty specific… is there a ticket open for that one? Pants uses PEX, which uses pip under the hood, so in the absence of any odd configuration, it should be the same speed as pip
@melodic-thailand-99227: it might be useful to open it here: https://github.com/pantsbuild/requirements-perf/issues?q=is%3Aissue , assuming that it repros there
w
@bitter-ability-32190: i don’t think so. he’s referring to the resolve itself, rather than consuming it
b
oh sorry, I meant related to the overall thread 🙂
If it helps, I've been spending some time on a "test suite" of sorts comparing various options with local/remote, hot/cold, with/without settings. It's mainly to help drive which settings to set on CI vs local, as well as show how Pants performs to our current baseline.
w
@bitter-ability-32190: ah… yea. the lint problem is the top issue on the board: i’ve retitled it to “Batch invokes of linters (and other dynamically sized process invokes)”
🙌 1
b
I can share the results, although the whole thing takes hours to run, so tweaks ain't easy 🙂
m
@witty-crayon-22786 our workflow is building a lock file with poetry and using poetry install to incrementally build the new virtual env. With pants the current workflow is poetry lock, and more time than the original lock to build the repository.pex (since pip is doing essentially the same work again, but less efficiently). What I would like is to tell pants here are the hashes of dependencies, no need to resolve anything, just install. Better yet, have pants understand that there’s only one package different and do an incremental work of building the repository.pex
👀 1
w
What I would like is to tell pants here are the hashes of dependencies, no need to resolve anything, just install.
got it. so attempt to take PIP out of the equation, essentially. i’m not sure of the degree to which that’s possible, but i’ll make sure it’s noted as part of the lockfile work that John is doing in PEX, and reference that in the performance list. thanks!
@melodic-thailand-99227: to follow up here: this has been discussed a bit on an (unrelated) PEX ticket, but one thing that is clear is that the move to Pants-native support for lockfiles will improve the time taken to both produce (because wheels won’t need to be installed) and consume (because consuming the lockfile to generate a set of wheels won’t need to resolve) the lockfile.
we have further ideas for how to improve the performance of subsetting, but for now, https://github.com/pantsbuild/pants/issues/13964 and https://github.com/pantsbuild/pants/issues/12314 will be the next improvements coming in this area