I’m trying to file a flaky test issue for failing ...
# generalh
hundreds-father-404
12/12/2018, 5:25 PMI’m trying to file a flaky test issue for failing contrib shard, but am unclear what test is actually failing. Someone mind helping me to parse the logs? https://api.travis-ci.org/v3/job/467047088/log.txt
hundreds-father-404
12/12/2018, 5:25 PMMaybe it’s this line:
contrib/avro/tests/java/org/pantsbuild/contrib/avro:avro[a
average-vr-56795
12/12/2018, 5:26 PMLooks like it was a network connectivity issue downloading the errorprone jar from maven?
r
red-balloon-89377
12/12/2018, 5:27 PMYou can Ctrl-F for
:: problems summary ::👌 1
red-balloon-89377
12/12/2018, 5:27 PMIt failed to fetch a jar
h
hundreds-father-404
12/12/2018, 5:28 PMOkay that’s helpful. Thank you both! And it looks like it failed to fetch the JAR during the
contrib/avro/tests/java/org/pantsbuild/contrib/avro:avrohundreds-father-404
12/12/2018, 5:28 PMDo we open
flaky-testr
red-balloon-89377
12/12/2018, 5:28 PMPlease do! It’s great to have them documented 🙂
👌 1
e
enough-analyst-54434
12/12/2018, 5:44 PMI'm not sure I agree with that! There is flaky expected - which is any test that uses network, and there is flaky unexpected - we wrote a bad test or are using non-deterministic data structures. The former are probably never going away, the latter we should always be able to squash.
enough-analyst-54434
12/12/2018, 5:45 PMSo from a label perspective, it seems to me network flakes just add noise to cleanup efforts. I advocate for a separate label or just not doing this.
👍 1
h
hundreds-father-404
12/12/2018, 5:46 PM@enough-analyst-54434 thoughts on a new tag
expected-flakya
aloof-angle-91616
12/12/2018, 5:46 PMwhat are your thoughts on adding retries to particularly flaky network requests?
👍 1
e
enough-analyst-54434
12/12/2018, 5:47 PMI don't believe there is such a thing as particularly flaky network requests. I would guess maven central, pypi and cargo central are all about as stable.
a
aloof-angle-91616
12/12/2018, 5:47 PMthere was that go download with the meta import tag that kept flaking? i need to page back into that
h
hundreds-father-404
12/12/2018, 5:48 PMI like retries. Could increase the number of timeouts we get, but better that than flaky imo.
Either way, I like the idea of still somehow cataloging in GitHub issues so that when you’re debugging the failure you don’t have to reproduce the effort some already put in to seeing if the failure is a flake
👍 1
a
aloof-angle-91616
12/12/2018, 5:48 PMin that case the answer might be to change what we are making requests for
e
enough-analyst-54434
12/12/2018, 5:48 PMLet me put it this way. Over the last few days I've fought flakes with a vengeance and the incidence of network to our bugs was maybe 1/10
enough-analyst-54434
12/12/2018, 5:48 PMI think our buggy tests are a much bigger problem
👍 3
a
aloof-angle-91616
12/12/2018, 5:48 PMok, that makes sense
aloof-angle-91616
12/12/2018, 5:49 PMi would feel more comfortable with a separate tag with much less priority? and tickets would only be made if there is relevant background or investigation, or an idea on how to fix them? i do 100% agree the buggy tests are the focus and this might help me to focus on that -- but i'm not wedded to the thought
e
enough-analyst-54434
12/12/2018, 5:50 PMI'm fine with a separate tag.
👍 1
h
hundreds-father-404
12/12/2018, 5:50 PMOkay, I’m thinking either a label
acceptable-flaky-testexpected-flaky-testa
aloof-angle-91616
12/12/2018, 5:51 PMci-network-failurelow-priority-flaky-teste
enough-analyst-54434
12/12/2018, 5:51 PMnetwork-flaky
👍 2
a
aloof-angle-91616
12/12/2018, 5:52 PM@hundreds-father-404 are you planning to make an issue with that tag for the above issue? a description on that tag that says it's lower priority might be helpful, not sure how github surfaces those descriptions in the ui
h
hundreds-father-404
12/12/2018, 5:54 PMYes, making the label right now
this is what I have for description:
Copy code
Flaky test resulting from network issues, implying the flake does not result from a bug in our code❤️ 1
a
aloof-angle-91616
12/12/2018, 5:54 PMthanks!
❤️ 1
e
enough-analyst-54434
12/12/2018, 5:55 PMThanks folks
💯 2
a
aloof-angle-91616
12/12/2018, 5:56 PMthanks for being the adult in the room and acknowledging the flakiness needs to be addressed! this increases my sanity tenfold
❤️ 1
h
hundreds-father-404
12/12/2018, 5:56 PMI’ll update the email thread to loop everyone in on this
👌 1
a
aloof-angle-91616
12/12/2018, 6:00 PMwe can update that label in the future when the sun begins to degrade and we have to guard against cosmic rays (pants will be around until then along with my brain in a jar (file))
😂 1
👌 1
3 Views