[webkit-dev] Skipping Flakey Tests

[David Levin]

If a test is flaky, it doesn't seem good to keep in the mix because it will
turn red for no reason and people will think a check in was bad.
oth, if a test is flaky but has enough logging in it to indicated an
underlying cause, then it may be worth keeping in the mix so that the root
issue may be determined.

imo, a good solution in this case is to improve the test to print out more
information to help track down the underlying bug.

dave

On Wed, Sep 23, 2009 at 11:23 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote:
>
> Alexey and I have been discussing if WebKit should add flakey tests to
> Skipped lists:
> https://bugs.webkit.org/show_bug.cgi?id=29322
>
> Alexey asked that I bring the discussion to a larger audience.
>
> Pros:
> - Buildbots stay green.
> - Red bots/tests means your change caused an error.
>
> Cons:
> - Skipped tests may be more likely to be forgotten (and thus never fixed).
>
> What does WebKit think?  Should we skip flakey tests which can't be
> resolved in a timely manner (and instead track issues via bugs) as
> WebKit policy?  Or should we reserve the skipped list for other
> conditions (like bugs in the OS)?
>
> Thoughts?
>
>
> I'm a little concerned about sweeping failures under the carpet, but on the
> other hand the buildbots are much less valuable if they are red too much.
> It's hard to think about this in the abstract. Do you have any concrete
> examples?
>
>  - Maciej
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090923/ef80fe8e/attachment.html>