[webkit-dev] Flaky test hit list

[Evan Martin]

On Wed, Sep 29, 2010 at 11:31 PM, Adam Barth <abarth at webkit.org> wrote:
> Tonight I wrote a new webkit-patch command for detecting flaky tests.

Are you familiar with the Chromium flakiness dashboard?
It provides a continuous summary of test results aggregated across
multiple builders, including views into the expected/actual results,
timing, logs, etc.

See for example
  http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=perf%2Ftyping-at-end-of-line.html
and click the "show results" link in the lower left to see the failure diff.

The front page, as I understand it, provides a list of the flakiest
tests in order.  (It also can cause your browser to melt, be
forewarned.)  And the json input the dashboard requires is already
produced by new-run-webkit-tests, so I think it'd be trivial to plug
the other WebKit builders into it.  (Given how small your hit list is,
I doubt the WebKit results would cause the dashboard to become as
heavyweight as the Chromium one is.)

PS: along those lines, here's an hierarchical view into test runtime
generated from the results json archived by those bots.  I hope to
hook this into the dashboard once I figure a few other things out.
  http://neugierig.org/software/chromium/tests/#Layout Demo
(Copy that URL including the space, sorry, it's a quick hack).