[webkit-dev] Skipping Flakey Tests

[David Levin]

I don't believe that the test was checked in a flaky state. It was solid for
a long time and then something happened...
I'll try to add more logging to this test this evening (after my turn at
helping chromium stay up to date with WebKit is over).

I'll ping Drew about the other test.

Dave


On Mon, Sep 28, 2009 at 4:40 PM, Eric Seidel <eric at webkit.org> wrote:

> On Fri, Sep 25, 2009 at 2:42 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> > I like Dave Levin's idea that the first action should be to instrument
> the
> > tests so we can find out why they intermittently fail. Especially if the
> > failure is reproducible on the bots but not on developer systems.
>
> I like this idea too.  I don't like the reality that flakey tests are
> a burden on all developers caused by one.
>
> > Using the
> > skip list should be a last resort, because that hides the failure instead
> of
> > helping us diangose the cause.
>
> I (respectfully) disagree.  I think we shouldn't be so afraid to skip
> tests.  We don't allow people to check in compiles which fail.  We
> don't allow people to check in tests which fail on other platforms
> (without skipping them) or on every other run.  Why should we allow
> people to check in tests which fail every 10 runs?  Or worse, why
> should we leave a known flakey test checked in/un-attended which fails
> every 10 runs?
>
> If we can't easily roll-out the failing tests (or the commit which
> cause them to start failing), we should skip them to keep the bots (a
> shared resource) green, so as not to block other work on the project.
> No?
>
> I very much like WebKit's "everyone is responsible for the whole
> project" culture, but I disagree that the burden of diagnosis should
> be on the person trying to make a completely unrelated checkin (as is
> the case when we leave flakey tests enabled in the tree).
>
> -eric
>
> p.s. I now have two "skipping flakey tests" changes up for review:
> https://bugs.webkit.org/show_bug.cgi?id=29322
> https://bugs.webkit.org/show_bug.cgi?id=29344
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090928/a74bd1a3/attachment.html>