[webkit-dev] More on test flakiness

[Julie Parent]

Given the feedback I've seen here, how about we do the following:

Specifically mark timing sensitive tests. If a marked tests fails, re-run
it.  If it passes the second time, consider it a normal pass and keep bots
green.  If it fails the second time, turn bots red.  The easiest way to mark
the tests seems to be to move them into a specific directory.

Any opposition or better ideas?

Thanks,
Julie

On Thu, Dec 3, 2009 at 6:49 AM, Gustavo Noronha Silva <gns at gnome.org> wrote:

> On Wed, 2009-12-02 at 14:51 -0800, Julie Parent wrote:
> > As Eric just said to me in person, another option is to just re-run
> > *any* failing test twice, and only turn tree red if it fails twice.
> > (Chromium just recently started doing this, and it has greatly
> > improved our tree greenness).  This obviously doesn't explicitly
> > identify timing dependent tests, but it solves the bigger issues that
> > flaky tests cause.
>
> But that would turn moot the point of the suggestion, I think. Having
> only tests that are expected to fail under special conditions be tested
> twice makes sense, but if a test that isn't expected to fail under
> special conditions fails, we should see that as a failure.
>
> To give you a bit of insight into this, in GTK+ we used to have tests
> that only failed when they were preceded by a specific test. This was
> very important information that would be lost if it was run after
> itself, and thus passed. The problems that caused this were missing
> support in DRT, and a couple of times real bugs that caused crashes.
>
> This is to say I think we would be better served by only running
> known-time-dependent tests twice.
>
> Thanks,
>
> --
> Gustavo Noronha Silva <gns at gnome.org>
> GNOME Project
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091204/7b84a5e9/attachment.html>