[Webkit-unassigned] [Bug 56675] New: Newly added layout tests shouldn't be reported as failing until after they consistently pass

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=56675

           Summary: Newly added layout tests shouldn't be reported as
                    failing until after they consistently pass
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: sfalken at apple.com


Enhancement suggestion for layout tests:
Newly added layout tests shouldn't be reported as failing until after they consistently pass.

When new tests are added, they are often flaky or fail completely on one or more platforms.
In order to keep the bots green, a fair amount of maintenance goes into resolving this situation.

Our current mechanism for addressing this is to manually add the flaky/failing test to the skip list, or add the failing results as expected.

It would be nicer if this was done automatically via:

- When a new test is added, it is run on all platforms
- But, when it fails, the failure doesn't turn the bots red
- Once the test passes on a specific platform X times in a row (10?) it would then be promoted (out of this probationary state)
- Any future failures would turn the bot red

Note that when the test is first checked in, we'd need to require that it pass on at least one platform, otherwise the test (or bugfix) isn't valid.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.