[webkit-dev] Skipping Flakey Tests

Darin Adler darin at apple.com
Fri Sep 25 13:59:10 PDT 2009

Green buildbots have a lot of value.

I think it’s worthwhile finding a way to have them even when there are  
test failures.

For predictable failures, the best approach is to land the expected  
failure as an expected result, and use a bug to track the fact that  
it’s wrong. To me this does seem a bit like “sweeping something under  
the rug”, a bug report is much easier to overlook than a red buildbot.  
We don’t have a great system for keeping track of the most important  

For tests that give intermittent and inconsistent results, the best we  
can currently do is to skip the test. I think it would make sense to  
instead allow multiple expected results. I gather that one of the  
tools used in the Chromium project has this concept and I think  
there’s no real reason not to add the concept to run-webkit-tests as  
long as we are conscientious about not using it when it’s not needed.  
And use a bug to track the fact that the test gives insufficient  
results. This has the same downsides as landing the expected failure  

For tests that have an adverse effect on other tests, the best we can  
currently do is to skip the test.

I think we are overusing the Skipped machinery at the moment for  
platform differences. I think in many cases it would be better to  
instead land an expected failure result. On the other hand, one really  
great thing about the Skipped file is that there’s a complete list in  
the file, allowing everyone to see the list. It makes a good to do  
list, probably better than just a list of bugs. This made Darin  
Fisher’s recent “why are so many tests skipped, lets fix it” message  

     -- Darin

More information about the webkit-dev mailing list