[Webkit-unassigned] [Bug 248553] New: Skip run-layout-tests-without-change step when first and second layout test run have only flaky failures

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=248553

            Bug ID: 248553
           Summary: Skip run-layout-tests-without-change step when first
                    and second layout test run have only flaky failures
           Product: WebKit
           Version: Other
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: aakash_jain at apple.com

In https://bugs.webkit.org/show_bug.cgi?id=204769, we changed ews behavior to ignore the flaky test failures. However, entire logic is applied at the very end of the build (after we run layout tests twice, and then compile-webkit and then run layout-tests without change).

However each of these steps might take significant time (especially important now that layout-test runtime has increased significantly in past few years). In order to make EWS more efficient, we can apply part of same logic just after re-run-layout-tests. The logic being, if both steps (layout-tests and re-run-layout-tests) completed and produced list of failures (without exiting early), and failures are completely different in both the runs (no single test consistently failed in both the runs), then we should simply ignore the failure and not blame the change being tested. 

Note that this is not at all a behavior/logic change, even currently we do the same thing in analyze-layout-tests-results step. Just that we apply the logic early and prevent extra steps of compiling-webkit and running layout-test without change.


For example in:
https://ews-build.webkit.org/#/builders/60/builds/44715

First run failures: fast/events/message-port-gc-after-removing-event-listener.html, fast/images/animated-heics-draw.html, http/wpt/service-workers/fetch-service-worker-preload.https.html
Second run failures: fast/forms/file/entries-api/image-no-transcode-open-panel.html, fast/images/animated-heics-verify.html

We still run compile-webkit-without-change and run-layout-tests-without-change

Since there were no consistent failures in first and second run, `compile-webkit-without-change` marked the build as successful. But we can reach that same conclusion without running layout-test on clean tree.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20221130/3e3bdf91/attachment.htm>