[webkit-dev] Iterating SunSpider

[Maciej Stachowiak]

On Jul 7, 2009, at 8:50 PM, Geoffrey Garen wrote:

>> I also don't buy your conclusion -- that if regular expressions  
>> account for 1% of JavaScript time on the Internet overall, they  
>> need not be optimized.
>>
>> I never said that.
>
> You said the regular expression test was "most likely... the least  
> relevant test" in SunSpider.
>
> You said implementors' choice to optimize regular expressions  
> because they were hot on SunSpider was "not what we want to  
> encourage."
>
> But maybe I misunderstood you. Do you think it was a good thing that  
> SunSpider encouraged optimization of regular expressions? If so, do  
> you think the same thing would have happened had SunSpider not used  
> summation in calculating its scores?

I suspect this line of questioning will not result in effective  
persuasion or useful information transfer. It comes off as kind of a  
gotcha question.

My understanding of Mike's position is this:

- The slowest test on the benchmark will become a focus of  
optimization regardless of scoring method (thus, I assume he does not  
really think regexp optimization efforts are an utter waste.)

- During the period when JS engines had most things much faster than  
the state of things when SunSpider first came out, but hadn't yet  
extensively optimized regexps, the test gave a misleading and  
potentially unfair picture of overall performance. And this is a  
condition that could happen again in the future.

I think this is a plausible position, but I don't entirely buy these  
arguments, and I don't think they outweigh the reasons we chose to use  
summation scoring. I think it's ultimately a judgment call, and unless  
we have new information to present, we don't need to drag out the  
conversation or call each to account on details of supporting arguments.

Regards,
Maciej