[webkit-dev] Skipping Flakey Tests

[Maciej Stachowiak]

On Sep 28, 2009, at 4:47 PM, David Levin wrote:

> I don't believe that the test was checked in a flaky state. It was  
> solid for a long time and then something happened...

What's "the test" in this context? The network / credentials one?

>
> I'll try to add more logging to this test this evening (after my  
> turn at helping chromium stay up to date with WebKit is over).
>
> I'll ping Drew about the other test.

It sounds like that test was always buggy, if Drew is right about the  
cause.

  - Macie

>
> Dave
>
>
> On Mon, Sep 28, 2009 at 4:40 PM, Eric Seidel <eric at webkit.org> wrote:
> On Fri, Sep 25, 2009 at 2:42 PM, Maciej Stachowiak <mjs at apple.com>  
> wrote:
> > I like Dave Levin's idea that the first action should be to  
> instrument the
> > tests so we can find out why they intermittently fail. Especially  
> if the
> > failure is reproducible on the bots but not on developer systems.
>
> I like this idea too.  I don't like the reality that flakey tests are
> a burden on all developers caused by one.
>
> > Using the
> > skip list should be a last resort, because that hides the failure  
> instead of
> > helping us diangose the cause.
>
> I (respectfully) disagree.  I think we shouldn't be so afraid to skip
> tests.  We don't allow people to check in compiles which fail.  We
> don't allow people to check in tests which fail on other platforms
> (without skipping them) or on every other run.  Why should we allow
> people to check in tests which fail every 10 runs?  Or worse, why
> should we leave a known flakey test checked in/un-attended which fails
> every 10 runs?
>
> If we can't easily roll-out the failing tests (or the commit which
> cause them to start failing), we should skip them to keep the bots (a
> shared resource) green, so as not to block other work on the project.
> No?
>
> I very much like WebKit's "everyone is responsible for the whole
> project" culture, but I disagree that the burden of diagnosis should
> be on the person trying to make a completely unrelated checkin (as is
> the case when we leave flakey tests enabled in the tree).
>
> -eric
>
> p.s. I now have two "skipping flakey tests" changes up for review:
> https://bugs.webkit.org/show_bug.cgi?id=29322
> https://bugs.webkit.org/show_bug.cgi?id=29344
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090928/7d8533bb/attachment.html>