[webkit-dev] Skipping Flakey Tests

[Eric Seidel]

Hum...  Discussion kinda died.  I'm interested in soliciting more
input, particularly from Apple folks.

Unless silence indicates that others agree with David Levin, Yaar and
Dimitri that we should skip flakey tests?

If that's the case, then I'd love a review on:
https://bugs.webkit.org/show_bug.cgi?id=29322
:)

Thanks for your time!

-eric

On Thu, Sep 24, 2009 at 1:40 PM, Eric Seidel <eric at webkit.org> wrote:
> I think the question is most interesting in the abstract as a question
> of general policy about flakey tests.  I think handling the
> circumstances of each bug is best left for discussion in the bug
> themselves.  That said, I've attached a list of recently filed bugs
> about flakey tests, per your request.
>
> -eric
>
> https://bugs.webkit.org/show_bug.cgi?id=29322 (mentioned in the original mail)
> https://bugs.webkit.org/show_bug.cgi?id=29505 (same root bug, I suspect)
> https://bugs.webkit.org/show_bug.cgi?id=29620 (being worked on,
> skipping not yet proposed)
> https://bugs.webkit.org/show_bug.cgi?id=28845 (OS bug, resulted in 2
> skips so far)
> https://bugs.webkit.org/show_bug.cgi?id=28624 (same OS bug, not yet skipped)
> https://bugs.webkit.org/show_bug.cgi?id=29035 (same OS bug, not yet skipped)
> https://bugs.webkit.org/show_bug.cgi?id=29154 (still being
> investigated, skipping not yet proposed)
>
> On Wed, Sep 23, 2009 at 11:23 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>> On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote:
>>
>> Alexey and I have been discussing if WebKit should add flakey tests to
>> Skipped lists:
>> https://bugs.webkit.org/show_bug.cgi?id=29322
>>
>> Alexey asked that I bring the discussion to a larger audience.
>>
>> Pros:
>> - Buildbots stay green.
>> - Red bots/tests means your change caused an error.
>>
>> Cons:
>> - Skipped tests may be more likely to be forgotten (and thus never fixed).
>>
>> What does WebKit think?  Should we skip flakey tests which can't be
>> resolved in a timely manner (and instead track issues via bugs) as
>> WebKit policy?  Or should we reserve the skipped list for other
>> conditions (like bugs in the OS)?
>>
>> Thoughts?
>>
>> I'm a little concerned about sweeping failures under the carpet, but on the
>> other hand the buildbots are much less valuable if they are red too much.
>> It's hard to think about this in the abstract. Do you have any concrete
>> examples?
>>  - Maciej
>>
>