I’m trying to debug a large memory increase between pants 2.6 and 2.13 (yeah, I know, but cannot bisect). Trying to lint (and check/typecheck) anything has pants using 120MB of memory on 2.6 but >1.2GB on 2.13 (this, btw, is just the resident size of pants, no external processes). I’ve tried using the

--stats-memory-summary

, but the sum of the sizes there is about 80MB. I’ve also tried using a memory profiler, but there are too many threads and calls into rust so I cannot really figure much out. What else can I do to figure it out?

You using pylint? At some point it started using transitive deps. But also if you're typechecking you'd already be doing that anyways 🤔

Yeah, if it was just pylint… But, we definitely want type checking. Now that you mention it,

resolve coarsened targets

is where the memory goes up.

Ah yeah makes sense then. CoarsenedTargets is your dependency graph, but in memory and very duplicitous

For my particular issue, if the memory could be freed after that, when running the lint stuff, that’d solve it 🙂

No idea 🙃 I think memoization is what's keeping it around. If you disable the daemon, or set a low memory limit it should get dropped with the process dropping

Yeah, we’re not using a daemon, but the whole thing goes above the 2GB memory of the container when building the PEX

We're exiting my area of knowledge here, but I would assume it has to do with # of targets, and how the grpah itself looks. The more interwoven, the more memory I would assume

So, this one I was testing with initially had 20 targets and like 15 files. Though, granted, it depends on bigger things. Let me try to find something really small 🙂

I could be wrong in that case it it might depend on other things 🙂

Yeah, so the actual targets aren’t the issue in this case, but the things that use a lot of memory do depend on big things. I tried with a smaller app, more self contianed, it’s 450MB (not great, not terrible 🙂 )