[webkit-dev] Iterating SunSpider

[Maciej Stachowiak]

On Jul 6, 2009, at 10:11 AM, Geoffrey Garen wrote:

>> So, what you end up with is after a couple of years, the slowest  
>> test in the suite is the most significant part of the score.   
>> Further, I'll predict that the slowest test will most likely be the  
>> least relevant test, because the truly important parts of JS  
>> engines were already optimized.  This has happened with Sunspider  
>> 0.9 - the regex portions of the test became the dominant factor,  
>> even though they were not nearly as prominent in the real world as  
>> they were in the benchmark.  This leads to implementors optimizing  
>> for the benchmark - and that is not what we want to encourage.
>
> How did you determine that regex performance is "not nearly as  
> prominent in the real world?"


For reference: in current JavaScriptCore, the one regexp-centric test  
is about 4.6% of the score by time. 3 of the string tests also spend  
their time in regexps, however, I think those are among the tests that  
most closely resemble what Web sites do. I believe the situation is  
roughly similar in other competitive JavaScript engines. This is  
probably not exactly proportionate but it doesn't dominate the test. I  
don't think any of this is a problem, unless one thinks the regexp  
improvements in Nitro, V8 and TraceMonkey were a waste of resources.

What I have seen happen is that numeric processing and especially  
integer math became a smaller and smaller proportion of the test,  
looking at the best publicly available engines over time. I think that  
turned out to be the case because math had much more room for  
optimization in naive implementations than, say, string processing.

Regards,
Maciej