[webkit-dev] Re: Automated tests (was: WebKit Project Goals)

[Darin Adler]

On Jul 26, 2007, at 11:13 AM, Andre-John Mas wrote:

> What should the requirements for the tool be, other than being  
> freely available?

I'm not sure these are requirements, but here are some of the things  
I'd like to see:

     - representative of real world performance
         - one way to make it a real-world test is to use content from  
real-world websites, but that's the rub because the copyright owner  
may not give us license to use it as we wish

     - easy to run in an automated fashion so we can post results on  
the buildbots
     - easy to run with performance analysis tools

     - possible to run in non-WebKit browsers too

     - no "axe to grind" -- this wouldn't be a test specifically  
designed to highlight a particular performance problem

     - doesn't change a lot over time -- it's hard to judge changes in  
performance of the engine if the test is changing too

There are other more subtle requirements. If we have too many  
different performance tests, they are no good because we can't run  
them all. If benchmarks are created by unaffiliated third parties,  
they are even more valuable because people won't assume they are  
"slanted".

     -- Darin