[Webkit-unassigned] [Bug 265066] New: [run-webkit-tests] Tests which fail and then crash on retry aren't flakey

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=265066

            Bug ID: 265066
           Summary: [run-webkit-tests] Tests which fail and then crash on
                    retry aren't flakey
           Product: WebKit
           Version: Safari Technology Preview
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: jbedard at apple.com

Our run-webkit-tests logic retries failed tests runs in some configurations. It will classify a test as "flakey" if the retry results don't match the original results, and EWS uses this to decide when to mark a change as having failed tests.

This logic isn't quite right, though, as demonstrated by https://github.com/WebKit/WebKit/pull/20624. When we say "flakey", we mean that the second test run matches an expected test result, not that the second test run is a different (but also failing) result.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20231117/5eeb97f1/attachment.htm>