[Webkit-unassigned] [Bug 219500] New: EWS for layout tests: On the retry steps only run the tests that failed on the first step.

bugzilla-daemon at webkit.org bugzilla-daemon at webkit.org
Thu Dec 3 11:53:17 PST 2020


https://bugs.webkit.org/show_bug.cgi?id=219500

            Bug ID: 219500
           Summary: EWS for layout tests: On the retry steps only run the
                    tests that failed on the first step.
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: clopez at igalia.com
                CC: aakash_jain at apple.com

When a patch that introduces failures the EWS for layout tests currently does this:

1.  step run-layout-tests: run all the tests with the pach
2.  step re-run-layout-tests: re-run all the tests with the pach
3.  step run-layout-tests-without-patch: re-run all the tests without the pach


Then it picks the tests that failed both in 1. and 2. and compares that with the tests that failed on 3.

tests_that_consistently_failed = first_results_failing_tests.intersection(second_results_failing_tests)
flaky_failures = first_results_failing_tests.union(second_results_failing_tests) - first_results_failing_tests.intersection(second_results_failing_tests)
failures_introduced_by_patch = tests_that_consistently_failed - clean_tree_results_failing_tests


The problem with this is the time that it takes to run the whole layout tests (between 30 minutes and 1 hour). So when a patch introduces new failures the EWS takes a lot to process.

This problem is specially relevant for WebKitGTK, where we are trying to add an EWS that runs layout tests but our tree is not always green, so even when the patch doesn't introduce new failures the EWS has to run the 3 steps to compare with the clean tree (which is not clean).


I think it would be idea to change this logic to on steps 2 and 3 to only run the tests that failed on the first step.

See example:

1.  step run-layout-tests: run all the tests with the pach (30 minutes)
    - Failed tests: {'fast/css/failure1.html', 'fast/css/failure2.html', 'fast/css/flaky_test.html'}

2.  step re-run-layout-tests: re-run only this 3 failures (1 minute)
    - Failed tests: {'fast/css/failure1.html', 'fast/css/failure2.html'}


3. step run-layout-tests-without-patch: re-run only the 3 failures (1 minute=
   - Failed tests: {'fast/css/flaky_test.html'}

result:

tests_that_consistently_failed =  {'fast/css/failure1.html', 'fast/css/failure2.html'}
flaky_failures =   {'fast/css/flaky_test.html'}
failures_introduced_by_patch = {'fast/css/failure1.html', 'fast/css/failure2.html'}


So the end result is the same but much faster as we only run once the whole layout test step.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20201203/1bc6085c/attachment-0001.htm>


More information about the webkit-unassigned mailing list