[webkit-dev] Skipping Flakey Tests

[Darin Adler]

On Oct 1, 2009, at 11:58 AM, Eric Seidel wrote:

> I think more you're identifying that there is a test hierarchy  
> problem here.  Chromium really wants to base its tests off of some  
> base "win" implementation, and then "win-apple", "win-chromium",  
> "win-cairo" results could derive from that, similar to how "mac" and  
> "mac-leopard", "mac-tiger", "mac-snowleopard" work.

Something like that would be excellent if this pattern turns up often.  
I don’t think we should make the change because of one test, but if it  
comes up a lot we definitely should.

> Back to the original topic:  I do however see flakey tests as  
> "endangering our testing strategy" because they provide false  
> negatives, and greatly reduce the value of the layout tests and  
> things which run the layout tests, like the buildbots or the commit- 
> bot.
>
> I also agree with Darin's earlier comment that WebKit needs  
> something like Chromium's multiple-expected results support so that  
> we can continue to run flakey tests, even if they're flakey instead  
> of having to resort to skipping them.  But for now, skipping is the  
> best we have, and I still encourage us to use it when necessary  
> instead of leaving layout tests flakey. :)

I agree on all of this.

Except that the two specific flakey tests we were discussing that got  
us started on this discussion were really serious bugs and it was  
really good to fix them rather than skipping them. After this  
experience, I now do share Alexey’s fear that if we had skipped them  
we would not have fixed the regression. Best, if possible, would have  
been to notice when they turned from reliable tests to flakey tests  
and rolled the change that made them flakey out.

     -- Darin