#general
- b
bored-art-40741
08/22/2015, 11:11 PMWell, that was the second problem - b
bored-art-40741
08/22/2015, 11:13 PMSwitching to zinc-only compilation (which we've wanted to do for a while) fixed the first problem, since it's not possible to schedule the jvm_compile group subtasks in non-deterministic order when there's only one of them - a
ancient-stone-50795
08/22/2015, 11:55 PMPatrick, there is an alternative Pants.isolated. ini in the root of the pants repo that might help. - b
bored-art-40741
08/23/2015, 12:00 AMAh interesting - b
bored-art-40741
08/23/2015, 12:00 AMI didn't realize there was flag to disable jmake - b
bored-art-40741
08/23/2015, 12:00 AMI ended up unregistering the jvm group task and re-registering it with only zinc - b
bored-art-40741
08/23/2015, 12:00 AM(We also don't use apt) - b
bored-art-40741
08/24/2015, 2:15 AMI think there's a rare situation that can hang a pants process, due I think to recent changes to nailgun client - b
bored-art-40741
08/24/2015, 2:16 AMI'm looking at a process where the root pants process is showing 100% CPU usage and is fully hung - b
bored-art-40741
08/24/2015, 2:16 AMpoking it with strace, I see
streaming in a tight looprecvfrom(11, "", 8096, 0, NULL, NULL) = 0 - b
bored-art-40741
08/24/2015, 2:16 AMI notice that 8096 is the number of bytes we try to read from the nailgun server - b
bored-art-40741
08/24/2015, 2:17 AMNot sure exactly how to debug further, I suspect that this is in
but I haven't gotten a proper traceNailgunSession.run - b
bored-art-40741
08/24/2015, 2:20 AMIt also seems to be consistently when invoking jar-tool - b
bored-art-40741
08/24/2015, 3:39 AMUnrelated (or maybe indirectly related?): about 5% of the time our runs of test.junit are failing withFATAL: VM exiting uncleanly - b
bored-art-40741
08/24/2015, 3:39 AMHas anyone else run into this? - b
bored-art-40741
08/24/2015, 3:39 AMIs there a way to get it to dump more debug info? - a
ancient-stone-50795
08/24/2015, 2:59 PM@bored-art-40741: w/r/t/
, We don’t see this error. It sounds to me like your VM might be getting killed (OOM handler?) or possibly its encountering a JVM bug and dumping core. We saw that a lot when running tests with 1.8.0_0 (but I don’t remember if it was this same error message)FATAL: VM exiting uncleanly - b
bored-art-40741
08/24/2015, 3:17 PMHmm, maybe our Xmx args aren't getting threaded to junit properly any more for some reason, I'll investigate that - b
bored-art-40741
08/24/2015, 3:18 PMThough generally when we OOM, we see the normal message for it - b
bored-art-40741
08/24/2015, 3:29 PMcc @rough-minister-58256 on the hanging stuff above. Looks like you've been working on that part of the codebase recently - b
bored-art-40741
08/24/2015, 3:29 PMLet me know if there's anything more I can provide, I think I still have a live (hung) process to poke at - b
bored-art-40741
08/24/2015, 3:30 PMAnd I can generate more with pants patches applied if necessary - w
witty-crayon-22786
08/24/2015, 4:33 PM@bored-art-40741: re VM exiting uncleanly: that means test code is `exit(N)`ing - w
witty-crayon-22786
08/24/2015, 4:33 PM@bored-art-40741: it has likely been doing that for a while, but more recent junitrunner protects against it. - w
witty-crayon-22786
08/24/2015, 4:34 PM(as an example, we had code internally that didn't stub out a shutdown timer thread, so it would non-deterministically kill unit tests with exit(0)... that used to exit cleanly.) - w
witty-crayon-22786
08/24/2015, 4:37 PM@happy-kitchen-89482: hah, good to know about the typer phase. i don't think it's strictly necessary to invoke the compiler a single time... they're just very reluctant to dip their toes into pants development. - w
witty-crayon-22786
08/24/2015, 4:39 PM@bored-art-40741: re: cache misses: i suspect an issue as well. my theory is that it is related to options fingerprinting consuming absolute paths inpants.ini - b
bored-art-40741
08/24/2015, 4:47 PMI figured out the cache misses eventually - b
bored-art-40741
08/24/2015, 4:47 PMIt was due to missing-deps=foo being rolled into the fingerprint - b
bored-art-40741
08/24/2015, 4:47 PMWe have warn locally and fatal on CI jobs that produce deployable artifacts