Splitting this into a new thread:

generate-lockfiles

takes 5 minutes per ‘thing’, how can I find out why?

what does the UI render while this is going on? assuming that the majority of the time is spent in a PEX subprocess, you’d need to profile the subprocess.

It’s slightly faster on 2.13.

good…

So from 10+ minutes to:

real    0m38.479s

ah, good.

I ran

./pants check ::

though, it’s been resolving coaresned targets for 40 seconds, then got warnings about

interpreter_constraints

that would be a different topic probably.

almost all our dependencies are pinned to a single version (99%), so not sure what that’s doing

the interpreter constraints thing is an error now

yea… that change is actually intended to improve performance

yeah, we just finished migrating from a mixed environment, nobody ever bothered to fix the

^3.7

vs

>3.7

stuff, I just removed everything basically 🙂

Hello. Hm, that's helpful. That is not the Pex subprocess. That is Pants itself before passing things to Pex, in figuring out your dependency graph. We would never expect it to take this long 😱 What was your performance like in Pants 2.6(?) when you do

./pants dependencies --transitive ::

? (To help us figure out if this is a regression)

I’m gonna try to wait for this to finish before I switch branches, sunk cost fallacy, I already waited 15 minutes… 😄

That definitely sounds like something is broken, like an infinite loop or pathological code path. Which we'd love to figure out, thank you for reporting this In the meantime to trying out 2.6, I'm curious: • how many files are in your repo? • Are you generally using one BUILD file per directory (what

tailor

does), or fewer BUILD files with

**

globs in the

sources

field? • are there import cycles between your files? • what does

./pants --concurrent roots | wc -l

in another terminal say? None of those would explain 15 minutes...but give us some info on the shape of your repository.

It’s not an infinite loop, it di finish.

Other questions that could be helpful: • what operating system you're using • how much RAM your machine has

23738 files (not all code, but the overwhelming majority), 262 BUILD files, though not one per directory, one per app/lib. I imagine there might be some, but I wouldn’t know offhand. 272 roots. It’s a 2019 mac, i7 I think, with 16GB of memory (I know 😞 ).

2.6 dependencies took 54 seconds

Certainly slower than we'd like—and we continue to make performance optimizations—but yeah, that's within the upper range I would expect for a repo that size. So, it sounds like somewhere between 2.6 and 2.13 there was a change that made this pathologically slow for you. It sounds like you were upgrading one version at a time. When do you first remember noticing how slow this was? I think there could be two approaches to figuring this out: 1. Bisect when things slowed down for you, or 2. use profiling to figure out on 2.13 where the slowdown is happening

I, unfortunately, didn’t commit anything between 2.6 and I think 2.10 😞 And even then, some things were horribly broken still.

some things were horribly broken still.

Meaning this performance slowdown, or other things?

But, as I said, last night when I ran it, on 2.13, it was fast

Okay, much much slower than we want for sure, and that's a bad regression from 2.6. But at least it terminates.. To clarify, is the extremely slow

generate-lockfiles

command on 2.13 also?

./pants dependencies --transitive ::

should trigger the same "Resolving targets" rule as

generate-lockfiles

was doing

I just ran it again (

./pants generate-lockfiles --resolve=pytest

), it’s been running for mroe than 2.5 minutes and doesn’t seem like it’s finishing. And, yeah, 2.13

Still stuck on the "Resolving transitive targets" step?

I wouldn’t say stuck, sometimes the timers do appear stuck, but not for very long.

@hundreds-father-404: if there are multiple entries for

Resolve transitive targets

, then it’s related to https://github.com/pantsbuild/pants/issues/11270 … but that should no longer be relevant under

generate-lockfiles

, I don’t think?

Screen Recording 2022-08-19 at 19.30.52.mov

would want to find where we’re making multiple TT calls for

generate-lockfiles

How can I help?

@ancient-france-42909: which backends do you have enabled?

would want to find where we’re making multiple TT calls for generate-lockfiles

@witty-crayon-22786 I suspect this has pathological behavior in their repo:

constraints = await _pytest_interpreter_constraints(python_setup)

all_tgts = await Get(AllTargets, AllTargetsRequest())
    transitive_targets_per_test = await MultiGet(
        Get(TransitiveTargets, TransitiveTargetsRequest([tgt.address]))
        for tgt in all_tgts
        if PythonTestFieldSet.is_applicable(tgt)
    )

But I don't understand why

dependencies --transitive ::

would be faster; iiuc it does the same type of

MultiGet

'pants.backend.python',
    "pants.backend.docker",
    "pants.backend.plugin_development",
    'pants.backend.python.lint.bandit',
    'pants.backend.python.lint.black',
    'pants.backend.python.lint.flake8',
    'pants.backend.python.lint.isort',
    'pants.backend.python.lint.pylint',
    "pants.backend.python.typecheck.mypy",
    "pants.backend.python.lint.docformatter",
    "pants.backend.python.mixed_interpreter_constraints",

And a custom one, which reminds me, I might’ve had it off last night when I was trying.

@hundreds-father-404: could be, yea. could maybe try resolve-by-resolve.

Stu, would it make sense to ask for

-ldebug

or anything like that to confirm where the rule is getting stuck?

I ran it without our plugin, it’s past 3 minutes now, so I don’t think it’s that.

could maybe try resolve-by-resolve.

George-Cristian, for context, I would expect

generate-lockfiles --resolve=isort

to be much faster, for example, because it does not need to consult your own code's interpreter constraints to figure out which to use for the lockfile. I suspect it's that interpreter constraints calculation for Pytest that is messing us up

Yes, that gets tot he generating lockfile stage instantly, 20s in total

Hey @witty-crayon-22786, one idea is https://github.com/pantsbuild/pants/issues/12652. I think it would allow us to short circuit most of the graph walking (Although that's a workaround -- this should not be so pathologically slow either way)

…er. wait a sec.

``` all_tgts = await Get(AllTargets, AllTargetsRequest())

transitive_targets_per_test = await MultiGet(

Get(TransitiveTargets, TransitiveTargetsRequest([tgt.address]))

for tgt in all_tgts

if PythonTestFieldSet.is_applicable(tgt)

)```

is no longer necessary after the IC deprecation, right?

Oh, that it can be solely looking at the "root" targets?

would still want to confirm that it’s actually that one, but

Yeah, that sounds correct to me! We can cherry-pick that to 2.13 because I think the deprecation was complete by then for ICs?

I’m going to have dinner, but will be back in a bit, if you want me to try stuff

@hundreds-father-404: correct: it’s an error in 2.13: https://github.com/pantsbuild/pants/pull/15531 is in there.

Thank you @ancient-france-42909 💜 If the above is all true, we have a path forward and can cherry-pick to 2.13! @witty-crayon-22786 I don't see where else this could be but the already quoted lines on ICs. No need for

-ldebug

to confirm

🤞

@ancient-france-42909 one command that would help us to confirm this fix:

./pants generate-lockfiles --resolve=black

, and make sure that

[black].interpreter_constraints

is not set. My change updates Pytest to use the same algorithm as Black was already using

Black was fast.

Awesome! I will have something out soon, and then we can do a release of 2.13 this weekend

btw, if you want me to test it, I can do it for a while tonight, but then not sure how much time I’ll have over the weekend. I mean, it’s not an issue as long as it’s related to generate-lockfiles, since I’m not going to do that anytime soon (someone else updated pytest today 😮), just wanted to make sure it makes it into the 2.13 release 🙂

Thank you! This is the PR and I will cherry-pick. Have a great weekend if we don't talk between then 🙂 https://github.com/pantsbuild/pants/pull/16591

is there an easy way to install from a branch? I was reading the docs earlier and it seemed to suggest you can use a SHA to install from master only

indeed, because we only build the wheels and push to S3 on

main

Okay, thanks! Now if you can make general dep resolving faster… 😄

Once this is merged, you'll be able to use PANTS_SHA https://github.com/pantsbuild/pants/pull/16596 We might have me do one related performance improvement on Monday, which would mean official release on Monday or Tuesday

Btw, that works great. Lockfiles are generated in 39 seconds for everything, vs 15+ minutes for one (though, I think it didn’t really scale linearly, I never tried)

Thanks for the update! That's great to hear 😄 I'm trying to get out the official release today, some CI issues

It’s okay, I don’t think I’ll be able to fix even half the issues I KNOW about this week, let alone more 🙂