Hi guys,
I hope you can help me with the issue we’...
# generalb
brief-flag-35586
11/05/2021, 6:27 PMHi guys,
I hope you can help me with the issue we’ve faced :)
We use pants 2.7.0 for our mono repo with about 20+ projects and since we switched from managed Github Actions runners to self-hosted we notice that pants is consuming a lot of memory to run python_tests.
When the limit was 4Gib the pods were often killed. Since then we’ve increased the limit to 8Gib and now it looks pretty stable, but sometimes the consumption goes as high as 85%=~7Gib and as the codebase is going to grow together with the tests, I have some concerns.
I believe we face a combination of https://github.com/pantsbuild/pants/issues/12662 and https://github.com/kubernetes/kubernetes/issues/43916 here, and there are a few workarounds around this issue, but maybe I’m missing/misconfigured something on the pants side as well.
I would appreciate any help :)
brief-flag-35586
11/05/2021, 6:27 PMCC @better-cricket-91243 🙂
🙌 1
brief-flag-35586
11/05/2021, 6:30 PMHere’s pants configuration we use for CI:
Copy code
[GLOBAL]
pantsd = false
process_execution_local_cache = false
remote_cache_read = true
remote_cache_write = true
remote_store_address = "<grpc://some.cache.domain>"
dynamic_ui = false
colors = true
process_execution_local_parallelism = 4
[python-setup]
resolver_jobs = 1
[test]
use_coverage = true
[stats]
log = truew
witty-crayon-22786
11/05/2021, 7:10 PMas the person assigned that ticket, i’d be remiss not to respond! heh
witty-crayon-22786
11/05/2021, 7:13 PMfrankly, we don’t have any tools in place for this, so the best bet is going to be a combination of native and python memory profiling tools
👍 1
witty-crayon-22786
11/05/2021, 7:13 PMand that is worth trying, in case you see any smoking guns.
witty-crayon-22786
11/05/2021, 7:14 PMto tackle it more deeply though, i expect that we’ll need to add support to pants for reporting the top-memory-consuming Graph nodes, followed by some optimization of whatever that ends up showing.
👍 1