[webkit-dev] Skipping Flakey Tests

[Darin Adler]

Green buildbots have a lot of value.

I think it’s worthwhile finding a way to have them even when there are  
test failures.

For predictable failures, the best approach is to land the expected  
failure as an expected result, and use a bug to track the fact that  
it’s wrong. To me this does seem a bit like “sweeping something under  
the rug”, a bug report is much easier to overlook than a red buildbot.  
We don’t have a great system for keeping track of the most important  
bugs.

For tests that give intermittent and inconsistent results, the best we  
can currently do is to skip the test. I think it would make sense to  
instead allow multiple expected results. I gather that one of the  
tools used in the Chromium project has this concept and I think  
there’s no real reason not to add the concept to run-webkit-tests as  
long as we are conscientious about not using it when it’s not needed.  
And use a bug to track the fact that the test gives insufficient  
results. This has the same downsides as landing the expected failure  
results.

For tests that have an adverse effect on other tests, the best we can  
currently do is to skip the test.

I think we are overusing the Skipped machinery at the moment for  
platform differences. I think in many cases it would be better to  
instead land an expected failure result. On the other hand, one really  
great thing about the Skipped file is that there’s a complete list in  
the file, allowing everyone to see the list. It makes a good to do  
list, probably better than just a list of bugs. This made Darin  
Fisher’s recent “why are so many tests skipped, lets fix it” message  
possible.

     -- Darin