[webkit-dev] Sunspider 0.9.1 preview

Maciej Stachowiak mjs at apple.com
Mon Dec 14 23:32:38 PST 2009

Hello folks,

Over the past few days I made some changes to SunSpider to address  
some of the more serious issues reported. I focused on only changes  
that seem to make a significant difference to fairness and validity,  
so for example I did not remove accidental access to global variables.  
I also made a small number of harness changes that do not affect  
results but fix flaws in the harness.

We are hesitant to change the SunSpider content or harness much at  
all, since it's been used for cross-version and cross-brwoser  
comparisons for so long. But these problems (many originally  
suggpointed out by Chrome or Mozilla folks) seemed important enough to  
address. Also, in addition to the patched content set, the original  
sunspider-0.9 content set is also available to run through the new  

The most important harness change is greatly reducing the time between  
tests (as sugested by Mike Belshe) to avoid the negative impact of  
power management on many systems (both Mac and Windows), and which are  
most apparent for very fast browsers.

I'm deliberately not posting this on the web site yet because I don't  
want a flood of gawkers testing their browser before enough people  
have had a chance to review and verify these changes.

Harness changes:

In-browser SunSpider suffers excessive penalty under power management

Enable Web-hosted version of SunSpider to handle multiple versions

Use JSON.parse instead of eval for Web-hosted SunSpider results  

Some Browser-hosted SunSpider files are not valid HTML5

Make sunspider-0.9.1 the default content set (both command-line and  

Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as  
originally posted):

SunSpider/tests/string-base64.js does not compute a valid base64  
encoded string

sunspider regexp-dna is inaccurate on firefox

Further changes I'm considering but am unsure about:
- Add correctness checking to all tests that don't use random numbers.
- Stop using array-like indexing of strings in the base64 test since  
that doesn't work in IE8 and lower; but it is a standard construct now  
(ES5), future IE will support it, and it's a useful thing to test.

Changes that probably won't be considered until a 2.0 version:
- Adding new tests to cover other areas.
- Rebalancing the runtime of the existing tests.
- Considering different scoring methodology such as bigger-is-better  
or geometric mean or the like.
- Removing use of random numbers from tests that do use them.


More information about the webkit-dev mailing list