[webkit-dev] Sunspider 0.9.1 preview
mjs at apple.com
Tue Dec 15 13:01:43 PST 2009
On Dec 15, 2009, at 12:30 PM, Mike Belshe wrote:
> [+cc John Resig since he's using this as part of dromaeo]
> Overall, sounds like good progress.
> A couple of ideas:
> - can we make it so that if you try to cut-and-paste comparisons
> of 0.9 to 0.9.1 results, it will say "these results are from a
> different version"?
Good idea. Filed <https://bugs.webkit.org/show_bug.cgi?id=32573>
> - can we make the version more prominent in the title?
I'll see if I can find a reasonable way to do so. <https://bugs.webkit.org/show_bug.cgi?id=32574
> - what would you think of reducing the setTimeout(..., 500) to
> something like setTimeout(..., 100)? This will cut the runtime of
> the test by ~80% :-)
I don't know if you noticed in my comments below, but the gap between
individual tests is now 10ms, there is only a one-time initial delay
of 500ms to give browsers a chance to recover from the effects of
loading the driver page.
Or are you talking about the initial 500ms delay? On my MacBook Pro,
in 64-bit Safari, each run of the test takes 370ms measured time, plus
260ms for the gaps between tests, and there are a total of 5 cycles.
So actual test time is 3780. So reducing the initial pause from 500ms
to 100ms would be a 10% improvement on total runtime of the benchmark.
I don't think that would be a meaningful difference. But I can look at
whether this number can be reduced without distorting the results.
> I'll volunteer to do any of these tasks this week if you want me to
> look at it.
In general help is welcome, but I think I can take care of the two
bugs cited above.
> On Mon, Dec 14, 2009 at 11:32 PM, Maciej Stachowiak <mjs at apple.com>
> Hello folks,
> Over the past few days I made some changes to SunSpider to address
> some of the more serious issues reported. I focused on only changes
> that seem to make a significant difference to fairness and validity,
> so for example I did not remove accidental access to global
> variables. I also made a small number of harness changes that do not
> affect results but fix flaws in the harness.
> We are hesitant to change the SunSpider content or harness much at
> all, since it's been used for cross-version and cross-brwoser
> comparisons for so long. But these problems (many originally
> suggpointed out by Chrome or Mozilla folks) seemed important enough
> to address. Also, in addition to the patched content set, the
> original sunspider-0.9 content set is also available to run through
> the new harness.
> The most important harness change is greatly reducing the time
> between tests (as sugested by Mike Belshe) to avoid the negative
> impact of power management on many systems (both Mac and Windows),
> and which are most apparent for very fast browsers.
> I'm deliberately not posting this on the web site yet because I
> don't want a flood of gawkers testing their browser before enough
> people have had a chance to review and verify these changes.
> Harness changes:
> In-browser SunSpider suffers excessive penalty under power management
> Enable Web-hosted version of SunSpider to handle multiple versions
> Use JSON.parse instead of eval for Web-hosted SunSpider results
> Some Browser-hosted SunSpider files are not valid HTML5
> Make sunspider-0.9.1 the default content set (both command-line and
> Content changes (in sunspider-0.9.1 suite only; sunspider-0.9 is as
> originally posted):
> SunSpider/tests/string-base64.js does not compute a valid base64
> encoded string
> sunspider regexp-dna is inaccurate on firefox
> Further changes I'm considering but am unsure about:
> - Add correctness checking to all tests that don't use random numbers.
> - Stop using array-like indexing of strings in the base64 test since
> that doesn't work in IE8 and lower; but it is a standard construct
> now (ES5), future IE will support it, and it's a useful thing to test.
> Changes that probably won't be considered until a 2.0 version:
> - Adding new tests to cover other areas.
> - Rebalancing the runtime of the existing tests.
> - Considering different scoring methodology such as bigger-is-better
> or geometric mean or the like.
> - Removing use of random numbers from tests that do use them.
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev