[webkit-dev] Iterating SunSpider

[Mike Belshe]

On Tue, Jul 7, 2009 at 4:20 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Jul 7, 2009, at 4:01 PM, Mike Belshe wrote:
>
>
>> I'd like benchmarks to:
>>    a) have meaning even as browsers change over time
>>    b) evolve.  as new areas of JS (or whatever) become important, the
>> benchmark should have facilities to include that.
>>
>> Fair?  Good? Bad?
>>
>
> I think we can't rule out the possibility of a benchmark becoming less
> meaningful over time. I do think that we should eventually produce a new and
> rebalanced set of test content. I think it's fair to say that time is
> approaching for SunSpider.


I certainly agree that updating the benchmark over time is necessary :-)


>
>
> In particular, I don't think geometric means are a magic bullet.


Yes, using a geometric mean does not mean that you never need to update the
test suite.  But it does give you a lot of mileage :-)  And I think its
closer to an "industry standard" than anything else (spec.org).



> When SunSpider was first created, regexps were a small proportion of the
> total execution in what were the fastest publicly available at the time.
> Eventually, everything else got much faster. So at some point, SunSpider
> said "it might be a good idea to quadruple the speed of regexp matching
> now". But if it used a geometric mean, it would always say it's a good idea
> to quadruple the speed of regexp matching, unless it omitted regexp tests
> entirely. From any starting point, and regardless of speed of other
> facilities, speeding up regexps by a factor of N would always show the same
> improvement in your overall score. SunSpider, on the other hand, was
> deliberately designed to highlight the area where an engine most needs
> improvement.


I don't think the optimization of regex would have been effected by using a
different scoring mechanism.  In both scoring methods, the score of the
slowest test is the best pick for improving your overall score.  So vendors
would still need to optimize it to keep up.


Mike



>
> I think the only real way to deal with this is to periodically revise and
> rebalance the benchmark.
>
> Regards,
> Maciej
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090707/e842d245/attachment.html>