https://pantsbuild.org/ logo
Join Slack
Powered by
My CI agent is killed with OOM when trying to run ...
# general
s
My CI agent is killed with OOM when trying to run 
pants lint
, how do I debug where the memory goes?
I have this config
Copy code
- name: PANTS_PANTSD
              value: "False"
            - name: PANTS_PROCESS_EXECUTION_LOCAL_ENABLE_NAILGUN
              value: "False"
            - name: PANTS_PROCESS_PER_CHILD_MEMORY_USAGE
              value: "512MiB"
            - name: PANTS_PROCESS_EXECUTION_LOCAL_PARALLELISM
              value: "6"
            - name: PANTS_PROCESS_TOTAL_CHILD_MEMORY_USAGE
              value: "6GiB"
            - name: PANTS_RULE_THREADS_CORE
              value: "4"
            - name: PANTS_RULE_THREADS_MAX
              value: "8"
and resources
Copy code
resources:
            limits:
              cpu: "12"
              memory: "24Gi"
            requests:
              cpu: "6"
              memory: "18Gi"
g
Do you have any idea if it's Pants itself or the processes Pants spawn that explodes? You can use 
--stats-memory-summary
on the pants CLI to get some information about Pants memory usage, but it wouldn't record what's happening in subprocs. I see you're using 
process-total-child-memory-usage
, which per docs only applies to JVM, so just want to confirm that you're aware of this given that you're also disabling Nailgun.
s
yeah, nailgun is disabled 
PANTS_PROCESS_EXECUTION_LOCAL_ENABLE_NAILGUN
Do you have any idea if it's Pants itself or the processes Pants spawn that explodes?
nope, i don't know yet
I'll try to look at some k8s pod metrics
I suspect the usage might have gone up after upgrade to pants 2.25.0rc0
g
yeah, nailgun is disabled PANTS_PROCESS_EXECUTION_LOCAL_ENABLE_NAILGUN
Yeah, so you disable nailgun (a JVM tool) but set 
PANTS_PROCESS_TOTAL_CHILD_MEMORY_USAGE
which is a JVM option. Just raising the point that you're both removing JVM tooling and configuring JVM settings. So if you don't use JVM that memory option has no effect.
I'll try to look at some k8s pod metrics
I can't share it but I've used a small psutil script that logs memory usage at 10Hz to stdout. Quite useful for debugging memory leaks when subprocs are involved.
s
So if you don't use JVM that memory option has no effect.
ah, I see, thanks
w
@gorgeous-winter-99296 Does Nailgun always run? Or only when the JVM backend is enabled?
g
It's started by default.
w
Yeah, that's what I thought - does anything non-JVM use it as well? I wondered about this while I was deep diving in the engine, but forgot to look further
g
Quick grep indicates it's only referenced by JVM stuff in backends.
w
Yeahh.... Okay, so that matches my notes from back in the day - but those were honestly educated guesses, not really much else
g
(Nailgun seems dead as a project anyways, it's been archived by Facebook)
Maybe worth ripping it out as a dependency. Not sure what buck2 does, if it even supports Java.
w
At the very least, limiting it's run to a JVM backend being enabled
5 Views
Open in Slack
PreviousNext
Powered by