[webkit-dev] Skipping Flakey Tests

[Eric Seidel]

On Fri, Sep 25, 2009 at 2:42 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> I like Dave Levin's idea that the first action should be to instrument the
> tests so we can find out why they intermittently fail. Especially if the
> failure is reproducible on the bots but not on developer systems.

I like this idea too.  I don't like the reality that flakey tests are
a burden on all developers caused by one.

> Using the
> skip list should be a last resort, because that hides the failure instead of
> helping us diangose the cause.

I (respectfully) disagree.  I think we shouldn't be so afraid to skip
tests.  We don't allow people to check in compiles which fail.  We
don't allow people to check in tests which fail on other platforms
(without skipping them) or on every other run.  Why should we allow
people to check in tests which fail every 10 runs?  Or worse, why
should we leave a known flakey test checked in/un-attended which fails
every 10 runs?

If we can't easily roll-out the failing tests (or the commit which
cause them to start failing), we should skip them to keep the bots (a
shared resource) green, so as not to block other work on the project.
No?

I very much like WebKit's "everyone is responsible for the whole
project" culture, but I disagree that the burden of diagnosis should
be on the person trying to make a completely unrelated checkin (as is
the case when we leave flakey tests enabled in the tree).

-eric

p.s. I now have two "skipping flakey tests" changes up for review:
https://bugs.webkit.org/show_bug.cgi?id=29322
https://bugs.webkit.org/show_bug.cgi?id=29344