[webkit-dev] Sunspider 0.9.1 preview

[Maciej Stachowiak]

Hello folks,

Over the past few days I made some changes to SunSpider to address  
some of the more serious issues reported. I focused on only changes  
that seem to make a significant difference to fairness and validity,  
so for example I did not remove accidental access to global variables.  
I also made a small number of harness changes that do not affect  
results but fix flaws in the harness.

We are hesitant to change the SunSpider content or harness much at  
all, since it's been used for cross-version and cross-brwoser  
comparisons for so long. But these problems (many originally  
suggpointed out by Chrome or Mozilla folks) seemed important enough to  
address. Also, in addition to the patched content set, the original  
sunspider-0.9 content set is also available to run through the new  
harness.

The most important harness change is greatly reducing the time between  
tests (as sugested by Mike Belshe) to avoid the negative impact of  
power management on many systems (both Mac and Windows), and which are  
most apparent for very fast browsers.

I'm deliberately not posting this on the web site yet because I don't  
want a flood of gawkers testing their browser before enough people  
have had a chance to review and verify these changes.


Harness changes:

In-browser SunSpider suffers excessive penalty under power management
https://bugs.webkit.org/show_bug.cgi?id=32505

Enable Web-hosted version of SunSpider to handle multiple versions
https://bugs.webkit.org/show_bug.cgi?id=32478

Use JSON.parse instead of eval for Web-hosted SunSpider results  
processing
https://bugs.webkit.org/show_bug.cgi?id=32490

Some Browser-hosted SunSpider files are not valid HTML5
https://bugs.webkit.org/show_bug.cgi?id=32536

Make sunspider-0.9.1 the default content set (both command-line and  
hosted)
https://bugs.webkit.org/show_bug.cgi?id=32537


Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as  
originally posted):

SunSpider/tests/string-base64.js does not compute a valid base64  
encoded string
https://bugs.webkit.org/show_bug.cgi?id=16806

sunspider regexp-dna is inaccurate on firefox
https://bugs.webkit.org/show_bug.cgi?id=18989



Further changes I'm considering but am unsure about:
- Add correctness checking to all tests that don't use random numbers.
- Stop using array-like indexing of strings in the base64 test since  
that doesn't work in IE8 and lower; but it is a standard construct now  
(ES5), future IE will support it, and it's a useful thing to test.

Changes that probably won't be considered until a 2.0 version:
- Adding new tests to cover other areas.
- Rebalancing the runtime of the existing tests.
- Considering different scoring methodology such as bigger-is-better  
or geometric mean or the like.
- Removing use of random numbers from tests that do use them.

Regards,
Maciej