[webkit-dev] Iterating SunSpider

[Maciej Stachowiak]

On Jul 7, 2009, at 4:01 PM, Mike Belshe wrote:

>
> I'd like benchmarks to:
>     a) have meaning even as browsers change over time
>     b) evolve.  as new areas of JS (or whatever) become important,  
> the benchmark should have facilities to include that.
>
> Fair?  Good? Bad?

I think we can't rule out the possibility of a benchmark becoming less  
meaningful over time. I do think that we should eventually produce a  
new and rebalanced set of test content. I think it's fair to say that  
time is approaching for SunSpider.

In particular, I don't think geometric means are a magic bullet. When  
SunSpider was first created, regexps were a small proportion of the  
total execution in what were the fastest publicly available at the  
time. Eventually, everything else got much faster. So at some point,  
SunSpider said "it might be a good idea to quadruple the speed of  
regexp matching now". But if it used a geometric mean, it would always  
say it's a good idea to quadruple the speed of regexp matching, unless  
it omitted regexp tests entirely. From any starting point, and  
regardless of speed of other facilities, speeding up regexps by a  
factor of N would always show the same improvement in your overall  
score. SunSpider, on the other hand, was deliberately designed to  
highlight the area where an engine most needs improvement.

I think the only real way to deal with this is to periodically revise  
and rebalance the benchmark.

Regards,
Maciej