[webkit-dev] Evolving the Sunspider Benchmark

[dsule at codeaurora.org]

I am wondering what the webkit community thoughts are with respect to
possibly evolving the sunspider benchmark.

I have worked with webkit based browsers and various javascript engines
shipping on mobile devices today and am seeing some very peculiar behavior
now starting to show up. As the various javascript engines have evolved
and become more and more efficient over the past few years, the time spent
executing individual sunspider subtests has decreased to a point that
javascript execution no longer seems to consuming most of the execution
time. Majority of the subtests have fallen to under 100ms in execution
times even on mobile devices. Rather, while the tests are executing, other
events like screen refreshes and paints seem to consume more time and even
tend to overshadow the end results reported by the testsuite at times.

Note, that my above observation is actually specific only to the sunspider
benchmark. In terms of real "page loading" timelines, javascript execution
still appears to be a significant factor.  Also, several of the other
javascript specific benchmarks like Google's V8suite and Mozilla's Kraken
benchmarks do not seem to show this same sort of limiting behavior since
they seem to be geared towards scripts running over longer timelines.

So, I was wondering if any others in the community were starting to notice
similar behavior and even more importantly if any effort is being planned
to try and evolve the sunspider benchmark to add any javascript issues
being noticed in todays newer websites into the benchmark so that it can
be kept up-to-date.

Thanks,
Dineel