[webkit-dev] Making performance tests in WebKit more useful

[Ryosuke Niwa]

On Fri, Mar 8, 2013 at 2:24 AM, Benjamin Poulain <benjamin at webkit.org>wrote:

> On Fri, Mar 8, 2013 at 2:15 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>> Three major problems with existing WebKit performance tests are:
>>
>>    1. There are too many tests to run
>>    2. Some tests have variances that are too high to be of any use
>>    3. Some tests are too specific to be of any general use
>>
>>
> If you find what differentiate the good tests from the bad tests, it would
> be good to create a wiki page explaining what to be careful about to make a
> good test.
>

Yeah. We need to spend more time analysing tests in categories 2 & 3.

 One thing that have bitten my ass in the past is the JIT being smart and
> removing some "dead" test code because it has no side effect. I am sure
> there are plenty of mistake like this to be avoided.
>

I was always curious about this. I wonder if there’s a good way to amortize
the effect so that we can measure relialistic number. The last thing I want
is to introduce a lot of measurement biases to lower the variance and end
up measuring something unrealistic.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130308/8cf0829d/attachment.html>