pantsbuild #general

@cuddly-architect-27013: rather than manually running unit tests, recommend creating a pull request from your personal repo into pantsbuild/pants, which will automatically run CI.

witty-crayon-22786

08/24/2015, 6:16 PM

@cuddly-architect-27013: merging

cuddly-architect-27013

08/24/2015, 6:17 PM

ok, thx @witty-crayon-22786

rough-minister-58256

08/24/2015, 6:20 PM

@bored-art-40741: essentially there would need to be some kind of nailgun kill or crash happening between the time pants launches the nailgun and the time the client sends a command and starts reading.

rough-minister-58256

08/24/2015, 6:20 PM

to create the behavior you’re describing - at least from what I’ve seen

rough-minister-58256

08/24/2015, 6:22 PM

pants would normally kill the nailgun if the fingerprint or java distribution changes; assuming thats not happening it could've been a crash

rough-minister-58256

08/24/2015, 6:22 PM

the nailgunclient could probably also be made smarter to detect when its trying to read from a dead nailgun (assuming thats the current case)

bored-art-40741

08/24/2015, 6:38 PM

hmmm

bored-art-40741

08/24/2015, 6:40 PM

So maybe this is an old bug that's just easier to trigger now because of some sort of nailgun timing?

bored-art-40741

08/24/2015, 6:44 PM

Copy code

def _read_chunk(self, buff):
    while len(buff) < self.HEADER_LENGTH:
      buff += self._sock.recv(self.BUFF_SIZE)

    payload_length, command = struct.unpack(self.HEADER_FMT, buff[:self.HEADER_LENGTH])
    buff = buff[self.HEADER_LENGTH:]
    while len(buff) < payload_length:
      buff += self._sock.recv(self.BUFF_SIZE)

    payload = buff[:payload_length]
    rest = buff[payload_length:]
    return command, payload, rest

bored-art-40741

08/24/2015, 6:44 PM

I'm suspicious of this

bored-art-40741

08/24/2015, 6:44 PM

self._sock.recv

blocking?

bored-art-40741

08/24/2015, 6:46 PM

Is all of the python examples, they appear to break the loop if the return from

self._sock.recv

is empty

bored-art-40741

08/24/2015, 6:49 PM

I'm strongly suspecting this is the problem

bored-art-40741

08/24/2015, 6:50 PM

The nailgun client zombies or dies and the socket isn't cleaned up for some reason. Then the client spins hard on a recv

bored-art-40741

08/24/2015, 6:52 PM

Oh but it is supposed to be a blocking call

bored-art-40741

08/24/2015, 6:54 PM

Yeah, that really confirms it

bored-art-40741

08/24/2015, 6:55 PM

That is a blocking call returning an empty list to indicate that the socket's client has shut down

bored-art-40741

08/24/2015, 7:19 PM

Tossed an exception in when

self._sock.recv

is empty, which I'm now very confident is handled incorrectly

bored-art-40741

08/24/2015, 7:19 PM

Going to see if I can repro now

bored-art-40741

08/24/2015, 7:22 PM

We'll still need to ask ourselves what it means if the socket disconnects in this way

bored-art-40741

08/24/2015, 7:22 PM

Is it an error that should cause pants to error out, or just log?

rough-minister-58256

08/24/2015, 7:28 PM

"Is all of the python examples, they appear to break the loop if the return from

self._sock.recv

is empty” <— yeah, I think this is a likely culprit in the “hangs forever doing socket reads from a dead nailgun” scenario and seems to be a long-standing bit of code. doesn’t explain why the nailgun died to begin with tho.

rough-minister-58256

08/24/2015, 7:29 PM

in that case I think it should either retry or fail as its essentially a “command execution failure” of the thing you were trying to execute on the nailgun.

rough-minister-58256

08/24/2015, 7:30 PM

since it seems rare, initially this could probably just throw a specific exception and retry could be handled later

bored-art-40741

08/24/2015, 7:33 PM

That's my thinking as well

bored-art-40741

08/24/2015, 7:33 PM

If I had to guess, I'd say that jar-tool is OOMing