Hey, so we recently (re?)-discovered that Pantsd c...
# development
h
Hey, so we recently (re?)-discovered that Pantsd can get OOM-killed, resulting in this confusing error message:
native_engine.NailgunClientException: Nailgun client error: "Nailgun client error: Client exited before the server's result could be returned."
That error message is cleaned up with https://github.com/pantsbuild/pants/pull/12107. But open question, should we be recommending Pantsd be used in CI? 🧵
We've had several reports of user CI breaking due to this error, which I imagine was OOM-killer given that CI often has lower RAM. Pants adding flakiness to CI is a mission critical issue Pro of disabling Pantsd: remove the risk of this flake Cons: - Will take longer, particularly in large repos w/ creating the build graph - Remote caching auth plugins can't leverage prior result, meaning most plugins will have to generate a new token every time - no automated garbage collection Users mucking around with
--pantsd-max-memory-usage
is an alternative, but not very simple. We could give recommend values based on CI runner?
Another detail: this is not always the result of Pantsd. Pants can crash if a single run is using too much memory, which we saw frequently with remote cache writes until John's recent fixes
I think I'm leaning towards keeping Pantsd, but us suggesting what to set the value to based on your CI provider
@fast-nail-55400 and I discussed - we're thinking have our docs on
pants.ci.toml
default to disabling Pantsd, but with an explanation of when it can be useful to enable + explaining tuning
--pantsd-max-memory-usage
thoughts @happy-kitchen-89482?
h
Hmm
I'm not sure
I thought John's changes made pantsd much less of a memory hog
h
For remote cache writes, indeed. But 2 users have reported seeing this error in CI and were not using remote caching