[webkit-dev] Making performance tests in WebKit more useful

[Ryosuke Niwa]

Hi,

Three major problems with existing WebKit performance tests are:

   1. There are too many tests to run
   2. Some tests have variances that are too high to be of any use
   3. Some tests are too specific to be of any general use


To address them, I’m going to segregate the tests into 3 tiers:

   1. Reliable tests that should be run on bots and locally when testing
   patches.
   2. Supplemental tests that could be ran optionally.
   3. Skipped tests.

In addition, I’m going to add a forth category between 1 and 2 for new
tests that have just been added since deciding whether a test is reliable
or not is hard unless we have some data.

This forth category is very important because while perf.webkit.org has an
ability to aggregate results for each suite (e.g. for the entire DOM) by
arbitary functions (e.g. arithmetic means, geometric means, etc…), letting
everyone add arbitary tests to any suite will undermine our ability to
monitor the results of reliable tests due to the added noise and means
skewed by new tests.  In an essense, we need a way to determine whether new
tests can be added to tier 1 “test suites”.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130308/520ede54/attachment.html>