[webkit-dev] Skipping Flakey Tests

[Maciej Stachowiak]

On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote:

> Alexey and I have been discussing if WebKit should add flakey tests to
> Skipped lists:
> https://bugs.webkit.org/show_bug.cgi?id=29322
>
> Alexey asked that I bring the discussion to a larger audience.
>
> Pros:
> - Buildbots stay green.
> - Red bots/tests means your change caused an error.
>
> Cons:
> - Skipped tests may be more likely to be forgotten (and thus never  
> fixed).
>
> What does WebKit think?  Should we skip flakey tests which can't be
> resolved in a timely manner (and instead track issues via bugs) as
> WebKit policy?  Or should we reserve the skipped list for other
> conditions (like bugs in the OS)?
>
> Thoughts?

I'm a little concerned about sweeping failures under the carpet, but  
on the other hand the buildbots are much less valuable if they are red  
too much. It's hard to think about this in the abstract. Do you have  
any concrete examples?

  - Maciej

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090923/3b668d57/attachment.html>