[Webkit-unassigned] [Bug 103839] New: FlakyTestReporter should be re-enabled and taught how to post patches

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=103839

           Summary: FlakyTestReporter should be re-enabled and taught how
                    to post patches
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: New Bugs
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: eric at webkit.org
                CC: abarth at webkit.org, ojan at chromium.org,
                    dpranke at chromium.org


FlakyTestReporter should be re-enabled and taught how to post patches

The FlakyTestReporter used to file nag-bugs, documenting tests that we saw flake on the EWS/commit-queue bots.

We disabled it because the nag-bugs weren't really that helpful.

However, now we could re-enable it and have it file bugs with patches to fix test-expectations to document the flake!

http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/tool/bot/flakytestreporter.py

Flaky tests are a big problem for the EWS/CQ bots.  The bots are smart enough to recover from occasionally flaky tests (up to 10 at a time), but:
1.  can't handle lots of flaky tests at once (there is a hard-coded limit before the bot gives up and assumes the tree is too unstable to determined anything about the patch)
2.  can't handle tests which flake super-often (the bot re-runs the tests to try and get repeatable results, very-flaky tests can trick the bot into thinking they're caused by the patch under testing and cause false-rejections).

The FlakyTestReporter tried solved these problems by nagging humans, but it could solve it more directly by posting a patch for r?/cq? with the needed lines for TestExpectations.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.