[webkit-dev] More on test flakiness

[Gustavo Noronha Silva]

On Wed, 2009-12-02 at 14:51 -0800, Julie Parent wrote:
> As Eric just said to me in person, another option is to just re-run
> *any* failing test twice, and only turn tree red if it fails twice.
> (Chromium just recently started doing this, and it has greatly
> improved our tree greenness).  This obviously doesn't explicitly
> identify timing dependent tests, but it solves the bigger issues that
> flaky tests cause.

But that would turn moot the point of the suggestion, I think. Having
only tests that are expected to fail under special conditions be tested
twice makes sense, but if a test that isn't expected to fail under
special conditions fails, we should see that as a failure.

To give you a bit of insight into this, in GTK+ we used to have tests
that only failed when they were preceded by a specific test. This was
very important information that would be lost if it was run after
itself, and thus passed. The problems that caused this were missing
support in DRT, and a couple of times real bugs that caused crashes.

This is to say I think we would be better served by only running
known-time-dependent tests twice.

Thanks,

-- 
Gustavo Noronha Silva <gns at gnome.org>
GNOME Project