[Webkit-unassigned] [Bug 219500] New: EWS for layout tests: On the retry steps only run the tests that failed on the first step.
bugzilla-daemon at webkit.org
bugzilla-daemon at webkit.org
Thu Dec 3 11:53:17 PST 2020
https://bugs.webkit.org/show_bug.cgi?id=219500
Bug ID: 219500
Summary: EWS for layout tests: On the retry steps only run the
tests that failed on the first step.
Product: WebKit
Version: WebKit Nightly Build
Hardware: Unspecified
OS: Unspecified
Status: NEW
Severity: Normal
Priority: P2
Component: Tools / Tests
Assignee: webkit-unassigned at lists.webkit.org
Reporter: clopez at igalia.com
CC: aakash_jain at apple.com
When a patch that introduces failures the EWS for layout tests currently does this:
1. step run-layout-tests: run all the tests with the pach
2. step re-run-layout-tests: re-run all the tests with the pach
3. step run-layout-tests-without-patch: re-run all the tests without the pach
Then it picks the tests that failed both in 1. and 2. and compares that with the tests that failed on 3.
tests_that_consistently_failed = first_results_failing_tests.intersection(second_results_failing_tests)
flaky_failures = first_results_failing_tests.union(second_results_failing_tests) - first_results_failing_tests.intersection(second_results_failing_tests)
failures_introduced_by_patch = tests_that_consistently_failed - clean_tree_results_failing_tests
The problem with this is the time that it takes to run the whole layout tests (between 30 minutes and 1 hour). So when a patch introduces new failures the EWS takes a lot to process.
This problem is specially relevant for WebKitGTK, where we are trying to add an EWS that runs layout tests but our tree is not always green, so even when the patch doesn't introduce new failures the EWS has to run the 3 steps to compare with the clean tree (which is not clean).
I think it would be idea to change this logic to on steps 2 and 3 to only run the tests that failed on the first step.
See example:
1. step run-layout-tests: run all the tests with the pach (30 minutes)
- Failed tests: {'fast/css/failure1.html', 'fast/css/failure2.html', 'fast/css/flaky_test.html'}
2. step re-run-layout-tests: re-run only this 3 failures (1 minute)
- Failed tests: {'fast/css/failure1.html', 'fast/css/failure2.html'}
3. step run-layout-tests-without-patch: re-run only the 3 failures (1 minute=
- Failed tests: {'fast/css/flaky_test.html'}
result:
tests_that_consistently_failed = {'fast/css/failure1.html', 'fast/css/failure2.html'}
flaky_failures = {'fast/css/flaky_test.html'}
failures_introduced_by_patch = {'fast/css/failure1.html', 'fast/css/failure2.html'}
So the end result is the same but much faster as we only run once the whole layout test step.
--
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20201203/1bc6085c/attachment-0001.htm>
More information about the webkit-unassigned
mailing list