https://pantsbuild.org/ logo
Join Slack
Powered by
Hey Pants Team, curious if there are any improveme...
# general
r
Hey Pants Team, curious if there are any improvements considered for 
Find targets from input specs
https://github.com/pantsbuild/pants/blob/bd1870586746954b2de7f9f0d842ef99df3a6b3a/src/python/pants/engine/internals/specs_rules.py#L268-L276 In a large repo it can take up a majority of the time just getting targets from specs. In our case I've seen up over 5 mins when we pass in a spec-file for just parsing dependencies in Python code. This is even with warm cache.
w
Which pants version is this using?
r
2.17.1
we are planning to upgrade soon
w
Would this be related to this discussion? https://github.com/pantsbuild/pants/discussions/17477
r
I don't think so, we use the rust parser. Mapping the specs to targets happens before that I thought
parsing the dependencies is really fast, just finding targets is extremely slow
h
We'd welcome any performance-improving PRs! There may be stuff that can be ported to rust there.
w
parsing the dependencies is really fast, just finding targets is extremely slow
Okay, good, this is what I was checking on - just wanted to confirm the rust parser was used and that it was working its sweet, sweet oxidized magic
🚀 1
r
Happy to help but not really sure where to begin...feels like something fundamental would have to change. The repository I'm working with is a good testing ground for this.
w
Is the repo OSS? Or can an anonymized version of it be used?
r
no this is a private internal repo
I'd imagine we can script up something to create a similar setup and start running pants to see these behaviors
I ran scc here's a general idea of what we are working with
Copy code
───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
Python                   48249   9255268   670488    269295  8315485     436501
HTML                     14850   3212185   601495    548083  2062607          0
JSON                      9100   7031303      489         0  7030814          0
Plain Text                4361    204353    21645         0   182708          0
Markdown                   841     65029    16017         0    49012          0
CSV                        497    330016      128         0   329888          0
XML Schema                 352     72338     2914         0    69424          0
XML                        299     86267        7         4    86256          0
Shell                      243     12402     1645      1180     9577       1415
YAML                       214     27803      280       178    27345          0
Smarty Template             46      1349      200         0     1149        225
ReStructuredText            28      2798      895         0     1903          0
TOML                        25      5418       88        40     5290          3
JavaScript                  21     13034      120        93    12821        160
CSS                         18      3730      678         2     3050          0
SQL                         16     33868        0         0    33868         17
gitignore                   15       300       52        48      200          0
BASH                        10       991      111        99      781        133
Expect                       7        85        2         0       83         11
Dockerfile                   4        59       10         4       45          6
Mako                         3        72       21         0       51          0
Stata                        3        45        0         0       45          0
Makefile                     2        43       11         7       25          6
Web Services Descri…         2      1600        0        27     1573          0
Docker ignore                1        23        0         0       23          0
Properties File              1        35        6         6       23          0
SVG                          1         1        0         0        1          0
TypeScript                   1       264       41        59      164         30
───────────────────────────────────────────────────────────────────────────────
Total                    79210  20360679  1317343    <tel:81912518224211|819125 18224211>     438507
───────────────────────────────────────────────────────────────────────────────
w
Yep, that’s a lot of files… How are you checking the time specifically for targets from specs?
r
TBH I just watch the logs from the console
it often goes over 300 seconds or more
👍 1
w
I wonder if this is the same issue I’m running into in one of my repos, wayyy smaller, but just a perf hiccup that seems out of place.
@rapid-bird-79300 Are you in any position time-wise to do some perf investigations? https://www.pantsbuild.org/2.19/docs/contributions/development/debugging-and-benchmarking#benchmarking-with-hyperfine
r
absolutely I'll get that going. Likely will share the results back here tomorrow if that's okay
w
Yeah! No rush, I’m just curious if we can isolate the problem, and maybe figure out if its code that needs to be rustified, or maybe there is something else to investigate
💯 1
r
I'm very curious what we find, hoping it leads to serious performance improvements for Pants. That would be epic for adoption.
❤️ 1
5 Views
Open in Slack
PreviousNext
Powered by