[Webkit-unassigned] [Bug 176104] New: [GTK] Improve the way unit test are run and the results reported

bugzilla-daemon at webkit.org bugzilla-daemon at webkit.org
Wed Aug 30 08:29:38 PDT 2017


https://bugs.webkit.org/show_bug.cgi?id=176104

            Bug ID: 176104
           Summary: [GTK] Improve the way unit test are run and the
                    results reported
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Keywords: Gtk
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: cgarcia at igalia.com
                CC: bugs-noreply at webkitgtk.org, lforschler at apple.com

There are several issues with the way unit tests are run by run-gtk-tests and also with the way results are reported:

 - The results summary only mentions the test binaries, not the actual test cases, so you always have to scroll up to find the actual test cases failing.
 - The number of reported failures is the number of test binaries that failed, so if a new test case fails for the same binary in a new revision, we won't notice it just looking at the number of failures.
 - We show detailed information about skipped test ins the results summary, which is just noise.
 - In the case of glib tests, when a test case times out, we finish the test suite, instead of continue with the rest of the test case like we do for normal failures or crashes. If a new test case fails after a test case that times out we will not notice until we fix or skip the test cases timing out.
 - In the case of glib tests, the timeout is aplied to the whole suite, instead of per test case, we have a hack to make it longer only for that. It has worked so far, but it doesn't scale, and it's an ugly hack.
 - It's not currently possible to detect flaky tests, because again, we know the binaries/suites that failed but not the actual test cases.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20170830/d58ba0bf/attachment.html>


More information about the webkit-unassigned mailing list