https://pantsbuild.org/ logo
#general
Title
# general
a

ancient-france-42909

09/10/2022, 6:37 PM
I’m trying to debug a large memory increase between pants 2.6 and 2.13 (yeah, I know, but cannot bisect). Trying to lint (and check/typecheck) anything has pants using 120MB of memory on 2.6 but >1.2GB on 2.13 (this, btw, is just the resident size of pants, no external processes). I’ve tried using the
--stats-memory-summary
, but the sum of the sizes there is about 80MB. I’ve also tried using a memory profiler, but there are too many threads and calls into rust so I cannot really figure much out. What else can I do to figure it out?
b

bitter-ability-32190

09/10/2022, 7:30 PM
You using pylint? At some point it started using transitive deps. But also if you're typechecking you'd already be doing that anyways 🤔
a

ancient-france-42909

09/10/2022, 7:33 PM
Yeah, if it was just pylint… But, we definitely want type checking. Now that you mention it,
resolve coarsened targets
is where the memory goes up.
b

bitter-ability-32190

09/10/2022, 7:35 PM
Ah yeah makes sense then. CoarsenedTargets is your dependency graph, but in memory and very duplicitous
Saves time, but eats memory
a

ancient-france-42909

09/10/2022, 7:36 PM
For my particular issue, if the memory could be freed after that, when running the lint stuff, that’d solve it 🙂
But, is there a way to work around this?
b

bitter-ability-32190

09/10/2022, 7:46 PM
No idea 🙃 I think memoization is what's keeping it around. If you disable the daemon, or set a low memory limit it should get dropped with the process dropping
a

ancient-france-42909

09/10/2022, 7:47 PM
Yeah, we’re not using a daemon, but the whole thing goes above the 2GB memory of the container when building the PEX
I mean, it’s going above the 2GB the workers on the CI have, otherwise, it wouldn’t really be an issue 🙂
Any idea on what I could look here? Is it a matter of the number of targets, or? In particular, I don’t know exactly what this scales with, a much bigger component uses 50MB more memory, at peak…
b

bitter-ability-32190

09/12/2022, 4:07 PM
We're exiting my area of knowledge here, but I would assume it has to do with # of targets, and how the grpah itself looks. The more interwoven, the more memory I would assume
a

ancient-france-42909

09/12/2022, 4:09 PM
So, this one I was testing with initially had 20 targets and like 15 files. Though, granted, it depends on bigger things. Let me try to find something really small 🙂
b

bitter-ability-32190

09/12/2022, 4:09 PM
I could be wrong in that case it it might depend on other things 🙂
a

ancient-france-42909

09/12/2022, 4:13 PM
Yeah, so the actual targets aren’t the issue in this case, but the things that use a lot of memory do depend on big things. I tried with a smaller app, more self contianed, it’s 450MB (not great, not terrible 🙂 )