https://pantsbuild.org/ logo
Powered by
#general
  • a

    ancient-stone-50795

    10/18/2016, 7:58 PM
    I wanted to compare the cmdline invocation to the junit-runner process
  • w

    witty-crayon-22786

    10/18/2016, 7:58 PM
    junit_run.py passes various args to the runner
  • w

    witty-crayon-22786

    10/18/2016, 7:58 PM
    yea, that
  • a

    ancient-stone-50795

    10/18/2016, 7:58 PM
    Well, I guess I don't need a failure to compare two command lines
  • w

    wide-energy-11069

    10/18/2016, 7:59 PM
    if it is a test dependency issue, you can use 
    --test-junit-test=<some specific failed test>
    to isolate the problem
  • a

    ancient-stone-50795

    10/18/2016, 8:15 PM
    I think I found a difference. Tests are sent to the command line in a different order.
  • w

    wide-energy-11069

    10/18/2016, 8:16 PM
    that does not surprise me. otherwise we wouldn’t been able to catch those flaky tests 🙂
  • a

    ancient-stone-50795

    10/18/2016, 8:18 PM
    I thought maybe they were sorted before but that doesn't seem the case. Its just that the tests are sent in a different arbitrary order.
  • w

    wide-energy-11069

    10/18/2016, 8:19 PM
    even between different runs within 1.0.14?
  • a

    ancient-stone-50795

    10/18/2016, 8:19 PM
    I did not back down to 1.0.14, I'd have to back down to 1.0.13.
  • a

    ancient-stone-50795

    10/18/2016, 8:19 PM
    But the tests aren't reliably failing on my mac - yet another sign that its just test flakiness
  • w

    witty-crayon-22786

    10/18/2016, 8:20 PM
    i could see an argument for sorting the tests, just to avoid shifts like this in the future.
  • w

    witty-crayon-22786

    10/18/2016, 8:21 PM
    but also, @wide-energy-11069 has suggested that an explicit "shuffled" mode might be handy to shake issues like this out, independently of core ci
  • a

    ancient-stone-50795

    10/18/2016, 8:27 PM
    Well, JUnit is supposed to always run the tests in the same order (at least, within a test class). That might be a change in behavior so let me back down to 1.0.13 and see
  • w

    wide-energy-11069

    10/18/2016, 8:28 PM
    does pants control the order by changing how to call junit runner, or does junit runner itself change the order?
  • w

    witty-crayon-22786

    10/18/2016, 8:32 PM
  • w

    witty-crayon-22786

    10/18/2016, 8:32 PM
    but it might lose it elsewhere.
  • l

    limited-country-20626

    10/18/2016, 9:03 PM
    I am still trying to get 1.2.0rc1 to fully pass our CIs. But this is the first time we have turned on the method parallelism so most of my issues so far are from that.
  • w

    witty-crayon-22786

    10/18/2016, 9:23 PM
    @limited-country-20626 : did you explicitly turn it on? iirc, it's not on by default
  • l

    limited-country-20626

    10/18/2016, 9:23 PM
    yep
  • w

    witty-crayon-22786

    10/18/2016, 9:24 PM
    brave!
  • l

    limited-country-20626

    10/18/2016, 9:24 PM
    ha - yeah!
  • l

    limited-country-20626

    10/18/2016, 9:24 PM
    I am selectively turning it off for um "legacy" tests with thread safety issues
  • l

    limited-country-20626

    10/18/2016, 9:24 PM
    kind of a skunk works thing but getting close to passing
  • a

    ancient-stone-50795

    10/19/2016, 1:44 AM
    FYI, backing down junit-runner to 1.0.13 doesn't change the failure pattern in our repo. Same projects are failing.
  • l

    limited-country-20626

    10/19/2016, 3:42 AM
    For a long time suspected that the batching was off - I had some junit tests that would periodically report as failing 2x but with different results. They were also some of the cruftiest tests in our repo (tldr hardcoded to a localhost port) so I ended up forcing TestSerial and moving on.
  • l

    limited-country-20626

    10/19/2016, 3:43 AM
    I even rolled back the runner to 1.0.03 for awhile but still saw the issues. <deleted a comment re: bootstrapping that would only affect us due to s/pom-resolve/ivy/>
  • w

    witty-crayon-22786

    10/19/2016, 7:14 PM
    @ancient-stone-50795 : so, we determined why we were seeing more failures, and it is (mostly) unrelated to the runner... we have a wrapper that allows tests marked 
    known_to_fail='$ticket'
    to fail if a particular flag is set
  • w

    witty-crayon-22786

    10/19/2016, 7:14 PM
    @ancient-stone-50795 : does it seem like the failures you are seeing are due to the ordering change then?
  • w

    witty-crayon-22786

    10/19/2016, 7:14 PM
    (and yea: we want to upstream the known_to_fail stuff)
1...151915201521...2420Latest