[webkit-dev] Iterating SunSpider

[George Staikos]

On 4-Jul-09, at 2:47 PM, Mike Belshe wrote:

> #2: Use of summing as a scoring mechanism is problematic
> Unfortunately, the sum-based scoring techniques do not withstand  
> the test of time as browsers improve.  When the benchmark was first  
> introduced, each test was equally weighted and reasonably large.   
> Over time, however, the test becomes dominated by the slowest tests  
> - basically the weighting of the individual tests is variable based  
> on the performance of the JS engine under test.  Today's engines  
> spend ~50% of their time on just string and date tests.  The other  
> tests are largely irrelevant at this point, and becoming less  
> relevant every day.  Eventually many of the tests will take near- 
> zero time, and the benchmark will have to be scrapped unless we  
> figure out a better way to score it.  Benchmarking research which  
> long pre-dates SunSpider confirms that geometric means provide a  
> better basis for comparison:  http://portal.acm.org/citation.cfm? 
> id=5673 Can future versions of the SunSpider driver be made so that  
> they won't become irrelevant over time?

    Actually this doesn't happen on all CPUs.  For example CPUs  
without FPU have very different results.  memory performance is also  
a big factor.

> #3: The SunSpider harness has a variance problem due to CPU power  
> savings modes.
> Because the test runs a tiny amount of Javascript (often under  
> 10ms) followed by a 500ms sleep, CPUs will go into power savings  
> modes between test runs.  This radically changes the performance  
> measurements and makes it so that comparison between two runs is  
> dependent on the user's power savings mode.  To demonstrate this,  
> run SunSpider on two machines- one with the Windows  
> "balanced" (default) setting for power, and then again with "high  
> performance".  It's easy to see skews of 30% between these two  
> modes.  I think we should change the test harness to avoid such  
> accidental effects.

    I've noticed this issue too.

--
George Staikos
Torch Mobile Inc.
http://www.torchmobile.com/