[Webkit-unassigned] [Bug 231999] [EWS][GTK][WPE] Add a new Class for running layout tests on the EWS for the GTK and WPE ports.

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=231999

Carlos Alberto Lopez Perez <clopez at igalia.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aakash_jain at apple.com,
                   |                            |ap at webkit.org,
                   |                            |jbedard at apple.com

--- Comment #1 from Carlos Alberto Lopez Perez <clopez at igalia.com> ---
We would like to add a new class to the EWS for running layout tests on the GTK and WPE ports.

Initially this class will be used for the GTK port, and later for WPE.
Mac/iOS ports will continue to use the current default EWS, so no behaviour change for them.

This new class is specially designed to:
  A. Work with a tree that is not always green (or even that is often quite red).
  B. To not report any false positive to the patch author.
  C. To allow patch authors to use this EWS to get new expectations for patches that need lot of new re-baselines (like a WPT import)

The very simplified logic of how this works is the following:
  1. Run layout tests with patch (abort early at 500 unexpected failures)
  2. Run layout tests with patch 10 times for each test that failed consistently (non-flaky) on step 1.
  3. Run layout tests without patch 10 times for each test that failed consistently (non-flaky) on step 2.

Then report to the patch author only the new consistent failures (tests that failed always with the patch and passed always without the patch, retrying 10 times)
Any flaky test found is only reported to the bot watchers.

For an explainer about why this is needed, and more details about the flow diagram and several design considerations, please check: https://people.igalia.com/clopez/wkbug/231999/explainer.html

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20211020/c8beaad3/attachment-0001.htm>