not...really

twitter hasn't had that problem

it's just painful

but we've grown

Would there be interest in me slowly adding

pylint

tests to the

pants

code base? I'm spotting a few errors not covered by tests with some

pylint

scans

@freezing-motorcycle-14222: talk to @mammoth-pharmacist-54215 - he recently added python linting support to pants itself. seems like the plan is to gradually increase the rules turned on in pants.ini in the pants repo.

Alright, I will. If he's working on this already I don't want to step on his toes

@freezing-motorcycle-14222: I’m gradually going through and enabling each of the tests, right now several of the tests are disabled in pants until I can get around to cleaning things up.

Then I'll focus on fixing specific errors that I find so that it's easier for you to enable it

Sounds good

@freezing-motorcycle-14222: you've seen https://rbcommons.com/s/twitter/r/2820/ , right?

would be good to enable an entire check rather than doing here-and-there fixes

@bored-art-40741, @limited-country-20626 : very interested in your profiling findings, as we have two folks working on it as well

Mateo is summarizing the results and noting the issues that are particularly pressing

Sneak preview:

(1) Heap usage by zinc is bad in the isolated strategy. It might be cache related, a memory leak, or the simple reality of compiling N leafy targets with huge classpaths all at the same time. If it's the latter, the only mitigation is really to be somewhat flexible about the number of workers that may actually run based on the size of the total analysis required in memory at any given time.

If it's the cache or leak related, then fixing the leak or just restarting the nailgun periodically might work.

The behavior in this situation is also really bad: it takes a long time for the thread to properly OOM, and sometimes it doesn't! It just trucks along for a very long time (we've confirmed the target doesn't actually take that long to compile in otherwise identical circumstances) and sometimes actually makes it out, or sometimes OOMs. It seems like there's just barely enough headroom for GC to keep scraping memory for a long time--but it's very counterintuitive, because profiling doesn't show us spending much time at all in GC in those situations. Might be some sort of weird thrashing behavior not blocking on GC, hard to say.

It's probably also worth investigating whether or not the nailgun itself is muddying this behavior. Not that we can really live without it, but we might be able to narrow down the actual weirdness of the failure mode.

"It just trucks along for a ​_very_​ long time (we've confirmed the target doesn't actually take that long to compile in otherwise identical circumstances) and sometimes actually makes it out" ... yea, this is what we've seen as well. in fact, we've never seen the oom

our CI killer usually gets to it first, and the restart causes it to resume

(2)

JvmCompile._register_vts

is noticeably slow on otherwise noop builds

Like a noop build goes from <5 seconds to >45

with or without the missing dep check?

Without

hm. that one doesn't make any sense to me, as the total product count should be identical

(between global and isolated)

not that i don't believe you: i just don't have a theory

We think the problem is the else clause in

JvmCompile.execute_chunk

that must always run though