[webkit-dev] Skipping Flakey Tests

Dimitri Glazkov dglazkov at chromium.org
Thu Sep 24 08:49:16 PDT 2009

I wonder if there's interest in the WebKit community to improve
granularity (and detail) with which you can document test
expectations, like we have done on Chromium:


This way, you could monitor known failures with the awesomeness that
is Layout Test Flakiness Dashboard:


Since we already have done this work on one port, we could definitely
extend this to all other ports.


On Wed, Sep 23, 2009 at 11:41 PM, David Levin <levin at google.com> wrote:
> If a test is flaky, it doesn't seem good to keep in the mix because it will
> turn red for no reason and people will think a check in was bad.
> oth, if a test is flaky but has enough logging in it to indicated an
> underlying cause, then it may be worth keeping in the mix so that the root
> issue may be determined.
> imo, a good solution in this case is to improve the test to print out more
> information to help track down the underlying bug.
> dave
> On Wed, Sep 23, 2009 at 11:23 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>> On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote:
>> Alexey and I have been discussing if WebKit should add flakey tests to
>> Skipped lists:
>> https://bugs.webkit.org/show_bug.cgi?id=29322
>> Alexey asked that I bring the discussion to a larger audience.
>> Pros:
>> - Buildbots stay green.
>> - Red bots/tests means your change caused an error.
>> Cons:
>> - Skipped tests may be more likely to be forgotten (and thus never fixed).
>> What does WebKit think?  Should we skip flakey tests which can't be
>> resolved in a timely manner (and instead track issues via bugs) as
>> WebKit policy?  Or should we reserve the skipped list for other
>> conditions (like bugs in the OS)?
>> Thoughts?
>> I'm a little concerned about sweeping failures under the carpet, but on
>> the other hand the buildbots are much less valuable if they are red too
>> much. It's hard to think about this in the abstract. Do you have any
>> concrete examples?
>>  - Maciej
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

More information about the webkit-dev mailing list