[webkit-dev] Sunspider 0.9.1 preview

[Mike Belshe]

On Tue, Dec 15, 2009 at 1:01 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Dec 15, 2009, at 12:30 PM, Mike Belshe wrote:
>
> [+cc John Resig since he's using this as part of dromaeo]
>
> Overall, sounds like good progress.
>
> A couple of ideas:
>    - can we make it so that if you try to cut-and-paste comparisons of 0.9
> to 0.9.1 results, it will say "these results are from a different version"?
>
>
> Good idea. Filed <https://bugs.webkit.org/show_bug.cgi?id=32573>
>
>    - can we make the version more prominent in the title?
>
>
> I'll see if I can find a reasonable way to do so. <
> https://bugs.webkit.org/show_bug.cgi?id=32574>
>
>    - what would you think of reducing the setTimeout(..., 500) to something
> like setTimeout(..., 100)?  This will cut the runtime of the test by ~80%
> :-)
>
>
> I don't know if you noticed in my comments below, but the gap between
> individual tests is now 10ms, there is only a one-time initial delay of
> 500ms to give browsers a chance to recover from the effects of loading the
> driver page.
>

Ok - I had checked the code, saw the 500ms setTimeout there, and then griped
:-)

This sounds great!




>
> Or are you talking about the initial 500ms delay? On my MacBook Pro, in
> 64-bit Safari, each run of the test takes 370ms measured time, plus 260ms
> for the gaps between tests, and there are a total of 5 cycles. So actual
> test time is 3780. So reducing the initial pause from 500ms to 100ms would
> be a 10% improvement on total runtime of the benchmark. I don't think that
> would be a meaningful difference. But I can look at whether this number can
> be reduced without distorting the results.
>

Agree - I don't care about that.

thanks,
Mike


>
>
> I'll volunteer to do any of these tasks this week if you want me to look at
> it.
>
>
> In general help is welcome, but I think I can take care of the two bugs
> cited above.
>
> Regards,
> Maciej
>
>
>
> Mike
>
>
> On Mon, Dec 14, 2009 at 11:32 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>>
>> Hello folks,
>>
>> Over the past few days I made some changes to SunSpider to address some of
>> the more serious issues reported. I focused on only changes that seem to
>> make a significant difference to fairness and validity, so for example I did
>> not remove accidental access to global variables. I also made a small number
>> of harness changes that do not affect results but fix flaws in the harness.
>>
>> We are hesitant to change the SunSpider content or harness much at all,
>> since it's been used for cross-version and cross-brwoser comparisons for so
>> long. But these problems (many originally suggpointed out by Chrome or
>> Mozilla folks) seemed important enough to address. Also, in addition to the
>> patched content set, the original sunspider-0.9 content set is also
>> available to run through the new harness.
>>
>> The most important harness change is greatly reducing the time between
>> tests (as sugested by Mike Belshe) to avoid the negative impact of power
>> management on many systems (both Mac and Windows), and which are most
>> apparent for very fast browsers.
>>
>> I'm deliberately not posting this on the web site yet because I don't want
>> a flood of gawkers testing their browser before enough people have had a
>> chance to review and verify these changes.
>>
>>
>> Harness changes:
>>
>> In-browser SunSpider suffers excessive penalty under power management
>> https://bugs.webkit.org/show_bug.cgi?id=32505
>>
>> Enable Web-hosted version of SunSpider to handle multiple versions
>> https://bugs.webkit.org/show_bug.cgi?id=32478
>>
>> Use JSON.parse instead of eval for Web-hosted SunSpider results processing
>> https://bugs.webkit.org/show_bug.cgi?id=32490
>>
>> Some Browser-hosted SunSpider files are not valid HTML5
>> https://bugs.webkit.org/show_bug.cgi?id=32536
>>
>> Make sunspider-0.9.1 the default content set (both command-line and
>> hosted)
>> https://bugs.webkit.org/show_bug.cgi?id=32537
>>
>>
>> Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as
>> originally posted):
>>
>> SunSpider/tests/string-base64.js does not compute a valid base64 encoded
>> string
>> https://bugs.webkit.org/show_bug.cgi?id=16806
>>
>> sunspider regexp-dna is inaccurate on firefox
>> https://bugs.webkit.org/show_bug.cgi?id=18989
>>
>>
>>
>> Further changes I'm considering but am unsure about:
>> - Add correctness checking to all tests that don't use random numbers.
>> - Stop using array-like indexing of strings in the base64 test since that
>> doesn't work in IE8 and lower; but it is a standard construct now (ES5),
>> future IE will support it, and it's a useful thing to test.
>>
>> Changes that probably won't be considered until a 2.0 version:
>> - Adding new tests to cover other areas.
>> - Rebalancing the runtime of the existing tests.
>> - Considering different scoring methodology such as bigger-is-better or
>> geometric mean or the like.
>> - Removing use of random numbers from tests that do use them.
>>
>> Regards,
>> Maciej
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091215/9c871e71/attachment.html>