[webkit-dev] Skipping Flakey Tests

[Dimitri Glazkov]

I wonder if there's interest in the WebKit community to improve
granularity (and detail) with which you can document test
expectations, like we have done on Chromium:

http://dev.chromium.org/developers/testing/webkit-layout-tests#TOC-Test-Expectations

This way, you could monitor known failures with the awesomeness that
is Layout Test Flakiness Dashboard:

http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html

Since we already have done this work on one port, we could definitely
extend this to all other ports.

:DG<

On Wed, Sep 23, 2009 at 11:41 PM, David Levin <levin at google.com> wrote:
> If a test is flaky, it doesn't seem good to keep in the mix because it will
> turn red for no reason and people will think a check in was bad.
> oth, if a test is flaky but has enough logging in it to indicated an
> underlying cause, then it may be worth keeping in the mix so that the root
> issue may be determined.
> imo, a good solution in this case is to improve the test to print out more
> information to help track down the underlying bug.
> dave
>
> On Wed, Sep 23, 2009 at 11:23 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>> On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote:
>>
>> Alexey and I have been discussing if WebKit should add flakey tests to
>> Skipped lists:
>> https://bugs.webkit.org/show_bug.cgi?id=29322
>>
>> Alexey asked that I bring the discussion to a larger audience.
>>
>> Pros:
>> - Buildbots stay green.
>> - Red bots/tests means your change caused an error.
>>
>> Cons:
>> - Skipped tests may be more likely to be forgotten (and thus never fixed).
>>
>> What does WebKit think?  Should we skip flakey tests which can't be
>> resolved in a timely manner (and instead track issues via bugs) as
>> WebKit policy?  Or should we reserve the skipped list for other
>> conditions (like bugs in the OS)?
>>
>> Thoughts?
>>
>> I'm a little concerned about sweeping failures under the carpet, but on
>> the other hand the buildbots are much less valuable if they are red too
>> much. It's hard to think about this in the abstract. Do you have any
>> concrete examples?
>>  - Maciej
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>