[webkit-dev] Sunspider 0.9.1 preview

[Maciej Stachowiak]

On Dec 15, 2009, at 12:30 PM, Mike Belshe wrote:

> [+cc John Resig since he's using this as part of dromaeo]
>
> Overall, sounds like good progress.
>
> A couple of ideas:
>    - can we make it so that if you try to cut-and-paste comparisons  
> of 0.9 to 0.9.1 results, it will say "these results are from a  
> different version"?

Good idea. Filed <https://bugs.webkit.org/show_bug.cgi?id=32573>

>    - can we make the version more prominent in the title?

I'll see if I can find a reasonable way to do so. <https://bugs.webkit.org/show_bug.cgi?id=32574 
 >

>    - what would you think of reducing the setTimeout(..., 500) to  
> something like setTimeout(..., 100)?  This will cut the runtime of  
> the test by ~80% :-)

I don't know if you noticed in my comments below, but the gap between  
individual tests is now 10ms, there is only a one-time initial delay  
of 500ms to give browsers a chance to recover from the effects of  
loading the driver page.

Or are you talking about the initial 500ms delay? On my MacBook Pro,  
in 64-bit Safari, each run of the test takes 370ms measured time, plus  
260ms for the gaps between tests, and there are a total of 5 cycles.  
So actual test time is 3780. So reducing the initial pause from 500ms  
to 100ms would be a 10% improvement on total runtime of the benchmark.  
I don't think that would be a meaningful difference. But I can look at  
whether this number can be reduced without distorting the results.

>
> I'll volunteer to do any of these tasks this week if you want me to  
> look at it.

In general help is welcome, but I think I can take care of the two  
bugs cited above.

Regards,
Maciej


>
> Mike
>
>
> On Mon, Dec 14, 2009 at 11:32 PM, Maciej Stachowiak <mjs at apple.com>  
> wrote:
>
> Hello folks,
>
> Over the past few days I made some changes to SunSpider to address  
> some of the more serious issues reported. I focused on only changes  
> that seem to make a significant difference to fairness and validity,  
> so for example I did not remove accidental access to global  
> variables. I also made a small number of harness changes that do not  
> affect results but fix flaws in the harness.
>
> We are hesitant to change the SunSpider content or harness much at  
> all, since it's been used for cross-version and cross-brwoser  
> comparisons for so long. But these problems (many originally  
> suggpointed out by Chrome or Mozilla folks) seemed important enough  
> to address. Also, in addition to the patched content set, the  
> original sunspider-0.9 content set is also available to run through  
> the new harness.
>
> The most important harness change is greatly reducing the time  
> between tests (as sugested by Mike Belshe) to avoid the negative  
> impact of power management on many systems (both Mac and Windows),  
> and which are most apparent for very fast browsers.
>
> I'm deliberately not posting this on the web site yet because I  
> don't want a flood of gawkers testing their browser before enough  
> people have had a chance to review and verify these changes.
>
>
> Harness changes:
>
> In-browser SunSpider suffers excessive penalty under power management
> https://bugs.webkit.org/show_bug.cgi?id=32505
>
> Enable Web-hosted version of SunSpider to handle multiple versions
> https://bugs.webkit.org/show_bug.cgi?id=32478
>
> Use JSON.parse instead of eval for Web-hosted SunSpider results  
> processing
> https://bugs.webkit.org/show_bug.cgi?id=32490
>
> Some Browser-hosted SunSpider files are not valid HTML5
> https://bugs.webkit.org/show_bug.cgi?id=32536
>
> Make sunspider-0.9.1 the default content set (both command-line and  
> hosted)
> https://bugs.webkit.org/show_bug.cgi?id=32537
>
>
> Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as  
> originally posted):
>
> SunSpider/tests/string-base64.js does not compute a valid base64  
> encoded string
> https://bugs.webkit.org/show_bug.cgi?id=16806
>
> sunspider regexp-dna is inaccurate on firefox
> https://bugs.webkit.org/show_bug.cgi?id=18989
>
>
>
> Further changes I'm considering but am unsure about:
> - Add correctness checking to all tests that don't use random numbers.
> - Stop using array-like indexing of strings in the base64 test since  
> that doesn't work in IE8 and lower; but it is a standard construct  
> now (ES5), future IE will support it, and it's a useful thing to test.
>
> Changes that probably won't be considered until a 2.0 version:
> - Adding new tests to cover other areas.
> - Rebalancing the runtime of the existing tests.
> - Considering different scoring methodology such as bigger-is-better  
> or geometric mean or the like.
> - Removing use of random numbers from tests that do use them.
>
> Regards,
> Maciej
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091215/1bc70233/attachment.html>