[webkit-dev] Skipping Flakey Tests

[Eric Seidel]

I think the question is most interesting in the abstract as a question
of general policy about flakey tests.  I think handling the
circumstances of each bug is best left for discussion in the bug
themselves.  That said, I've attached a list of recently filed bugs
about flakey tests, per your request.

-eric

https://bugs.webkit.org/show_bug.cgi?id=29322 (mentioned in the original mail)
https://bugs.webkit.org/show_bug.cgi?id=29505 (same root bug, I suspect)
https://bugs.webkit.org/show_bug.cgi?id=29620 (being worked on,
skipping not yet proposed)
https://bugs.webkit.org/show_bug.cgi?id=28845 (OS bug, resulted in 2
skips so far)
https://bugs.webkit.org/show_bug.cgi?id=28624 (same OS bug, not yet skipped)
https://bugs.webkit.org/show_bug.cgi?id=29035 (same OS bug, not yet skipped)
https://bugs.webkit.org/show_bug.cgi?id=29154 (still being
investigated, skipping not yet proposed)

On Wed, Sep 23, 2009 at 11:23 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
> On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote:
>
> Alexey and I have been discussing if WebKit should add flakey tests to
> Skipped lists:
> https://bugs.webkit.org/show_bug.cgi?id=29322
>
> Alexey asked that I bring the discussion to a larger audience.
>
> Pros:
> - Buildbots stay green.
> - Red bots/tests means your change caused an error.
>
> Cons:
> - Skipped tests may be more likely to be forgotten (and thus never fixed).
>
> What does WebKit think?  Should we skip flakey tests which can't be
> resolved in a timely manner (and instead track issues via bugs) as
> WebKit policy?  Or should we reserve the skipped list for other
> conditions (like bugs in the OS)?
>
> Thoughts?
>
> I'm a little concerned about sweeping failures under the carpet, but on the
> other hand the buildbots are much less valuable if they are red too much.
> It's hard to think about this in the abstract. Do you have any concrete
> examples?
>  - Maciej
>