What would be the cause for pants 1.30 suddenly fa...
# generalb
brainy-solstice-27042
08/08/2022, 9:26 PMWhat would be the cause for pants 1.30 suddenly failing with this stacktrace? We suddenly encountered this last week and after clearing out the cache, we still run into this issue on our Team City hosts, but can't replicate locally. Mainly trying to understand why pants would fail this way.
e
enough-analyst-54434
08/08/2022, 9:34 PMfailed with -9kill -9 <pid>enough-analyst-54434
08/08/2022, 9:35 PMSince you blew away caches, the relation is probably a bunch more work to do than normal, and that fills up memory.
enough-analyst-54434
08/08/2022, 9:36 PMLikely the only way out is to add memory if you can to the TeamCity nodes running CI.
b
brainy-solstice-27042
08/08/2022, 9:42 PMAh, that would make sense then. I'll try bumping that memory up to see if that solves the issue
brainy-solstice-27042
08/10/2022, 2:43 PMI gave pants more room to work, increasing the container it runs in from 2GB of RAM to 4 GB however that still didn't fix the problem. Some of the time the build for a specific service works, other times it doesn't with the same error above.
e
enough-analyst-54434
08/10/2022, 2:47 PMYeah, the nature of the OOMKiller is somewhat obscure. It uses a not-straight-forward algorithm to decide what processes to pick for killing. The short of it is though that you're still memory constrained but closer to the edge. Before you were way over the edge apparently.
h
hundreds-father-404
08/10/2022, 2:49 PMIt may be worth experimeting with these: https://www.pantsbuild.org/docs/using-pants-in-ci#tuning-resource-consumption-advanced
e
enough-analyst-54434
08/10/2022, 2:51 PMThis is Pants 1.30
👍 1
enough-analyst-54434
08/10/2022, 2:54 PMYou might be able to play with
--pantsd-max-memory-usageb
brainy-solstice-27042
08/10/2022, 3:05 PMI'll try the max memory usage option, that may be what we need to prevent pants from growing without bounds. Although the docs day it defaults to 1 GB, so that's curious why we're still failing. Either way, I'll try it out
e
enough-analyst-54434
08/10/2022, 3:35 PMThe v1 docs link I gave above is not anchor-friendly, but if you expand global options you can find it's actualy 4GB in v1:
b
brainy-solstice-27042
08/10/2022, 3:38 PMOh, that makes sense then, thanks!
5 Views