[webkit-dev] Sunspider 0.9.1 preview

[Mike Belshe]

[+cc John Resig since he's using this as part of dromaeo]

Overall, sounds like good progress.

A couple of ideas:
   - can we make it so that if you try to cut-and-paste comparisons of 0.9
to 0.9.1 results, it will say "these results are from a different version"?
   - can we make the version more prominent in the title?
   - what would you think of reducing the setTimeout(..., 500) to something
like setTimeout(..., 100)?  This will cut the runtime of the test by ~80%
:-)

I'll volunteer to do any of these tasks this week if you want me to look at
it.

Mike


On Mon, Dec 14, 2009 at 11:32 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> Hello folks,
>
> Over the past few days I made some changes to SunSpider to address some of
> the more serious issues reported. I focused on only changes that seem to
> make a significant difference to fairness and validity, so for example I did
> not remove accidental access to global variables. I also made a small number
> of harness changes that do not affect results but fix flaws in the harness.
>
> We are hesitant to change the SunSpider content or harness much at all,
> since it's been used for cross-version and cross-brwoser comparisons for so
> long. But these problems (many originally suggpointed out by Chrome or
> Mozilla folks) seemed important enough to address. Also, in addition to the
> patched content set, the original sunspider-0.9 content set is also
> available to run through the new harness.
>
> The most important harness change is greatly reducing the time between
> tests (as sugested by Mike Belshe) to avoid the negative impact of power
> management on many systems (both Mac and Windows), and which are most
> apparent for very fast browsers.
>
> I'm deliberately not posting this on the web site yet because I don't want
> a flood of gawkers testing their browser before enough people have had a
> chance to review and verify these changes.
>
>
> Harness changes:
>
> In-browser SunSpider suffers excessive penalty under power management
> https://bugs.webkit.org/show_bug.cgi?id=32505
>
> Enable Web-hosted version of SunSpider to handle multiple versions
> https://bugs.webkit.org/show_bug.cgi?id=32478
>
> Use JSON.parse instead of eval for Web-hosted SunSpider results processing
> https://bugs.webkit.org/show_bug.cgi?id=32490
>
> Some Browser-hosted SunSpider files are not valid HTML5
> https://bugs.webkit.org/show_bug.cgi?id=32536
>
> Make sunspider-0.9.1 the default content set (both command-line and hosted)
> https://bugs.webkit.org/show_bug.cgi?id=32537
>
>
> Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as
> originally posted):
>
> SunSpider/tests/string-base64.js does not compute a valid base64 encoded
> string
> https://bugs.webkit.org/show_bug.cgi?id=16806
>
> sunspider regexp-dna is inaccurate on firefox
> https://bugs.webkit.org/show_bug.cgi?id=18989
>
>
>
> Further changes I'm considering but am unsure about:
> - Add correctness checking to all tests that don't use random numbers.
> - Stop using array-like indexing of strings in the base64 test since that
> doesn't work in IE8 and lower; but it is a standard construct now (ES5),
> future IE will support it, and it's a useful thing to test.
>
> Changes that probably won't be considered until a 2.0 version:
> - Adding new tests to cover other areas.
> - Rebalancing the runtime of the existing tests.
> - Considering different scoring methodology such as bigger-is-better or
> geometric mean or the like.
> - Removing use of random numbers from tests that do use them.
>
> Regards,
> Maciej
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091215/32e1f963/attachment.html>