[webkit-dev] More on test flakiness
jparent+webkit at gmail.com
Fri Dec 4 15:12:13 PST 2009
Given the feedback I've seen here, how about we do the following:
Specifically mark timing sensitive tests. If a marked tests fails, re-run
it. If it passes the second time, consider it a normal pass and keep bots
green. If it fails the second time, turn bots red. The easiest way to mark
the tests seems to be to move them into a specific directory.
Any opposition or better ideas?
On Thu, Dec 3, 2009 at 6:49 AM, Gustavo Noronha Silva <gns at gnome.org> wrote:
> On Wed, 2009-12-02 at 14:51 -0800, Julie Parent wrote:
> > As Eric just said to me in person, another option is to just re-run
> > *any* failing test twice, and only turn tree red if it fails twice.
> > (Chromium just recently started doing this, and it has greatly
> > improved our tree greenness). This obviously doesn't explicitly
> > identify timing dependent tests, but it solves the bigger issues that
> > flaky tests cause.
> But that would turn moot the point of the suggestion, I think. Having
> only tests that are expected to fail under special conditions be tested
> twice makes sense, but if a test that isn't expected to fail under
> special conditions fails, we should see that as a failure.
> To give you a bit of insight into this, in GTK+ we used to have tests
> that only failed when they were preceded by a specific test. This was
> very important information that would be lost if it was run after
> itself, and thus passed. The problems that caused this were missing
> support in DRT, and a couple of times real bugs that caused crashes.
> This is to say I think we would be better served by only running
> known-time-dependent tests twice.
> Gustavo Noronha Silva <gns at gnome.org>
> GNOME Project
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev