[webkit-dev] Skipping Flakey Tests

[Darin Adler]

On Oct 1, 2009, at 11:41 AM, Drew Wilson wrote:

> I don't have an opinion about flakey tests, but flat-out-busted  
> tests should get skipped. Any thoughts/objections?

I object.

If a test fails on some platforms and succeeds on others, we should  
have the success result checked in as the default case, and the  
failure as an exception. And we should structure test results and  
exceptions so that it’s easy to get the expected failure on the right  
platforms and success on others. Your story about a slight  
inconvenience because a test failed on the base Windows WebKit but  
succeeded on the Chromium WebKit does not seem like a reason to change  
this!

Skipping the test does not seem like a good thing to do for the long  
term health of the project. It is good to exercise all the other code  
each test covers and also to notice when a test result gets even worse  
or gets better when a seemingly unrelated change is made.

I think we should skip only tests that endanger the testing strategy  
because they are super-slow, crash, or adversely affect other tests in  
some way.

     -- Darin