[webkit-dev] Skipping Flakey Tests

[Eric Seidel]

On Fri, Sep 25, 2009 at 1:59 PM, Darin Adler <darin at apple.com> wrote:
> For tests that give intermittent and inconsistent results, the best we can
> currently do is to skip the test. I think it would make sense to instead
> allow multiple expected results. I gather that one of the tools used in the
> Chromium project has this concept and I think there’s no real reason not to
> add the concept to run-webkit-tests as long as we are conscientious about
> not using it when it’s not needed.

Not to derail the discussion, but to provide context for Darin's reply:

Yes, Chromium's version of run-webkit-tests (called
run_webkit_tests.py) has multiple-expected-results support (along with
running tests in parallel and other goodness).  But it's missing a
bunch of the nifty flags WebKit's run-webkit-tests has.  My hope is to
eventually unify them, which we've filed some bugs on:
http://code.google.com/p/chromium/issues/detail?id=23099
https://bugs.webkit.org/show_bug.cgi?id=10906

-eric

On Fri, Sep 25, 2009 at 1:59 PM, Darin Adler <darin at apple.com> wrote:
> Green buildbots have a lot of value.
>
> I think it’s worthwhile finding a way to have them even when there are test
> failures.
>
> For predictable failures, the best approach is to land the expected failure
> as an expected result, and use a bug to track the fact that it’s wrong. To
> me this does seem a bit like “sweeping something under the rug”, a bug
> report is much easier to overlook than a red buildbot. We don’t have a great
> system for keeping track of the most important bugs.
>
> For tests that give intermittent and inconsistent results, the best we can
> currently do is to skip the test. I think it would make sense to instead
> allow multiple expected results. I gather that one of the tools used in the
> Chromium project has this concept and I think there’s no real reason not to
> add the concept to run-webkit-tests as long as we are conscientious about
> not using it when it’s not needed. And use a bug to track the fact that the
> test gives insufficient results. This has the same downsides as landing the
> expected failure results.
>
> For tests that have an adverse effect on other tests, the best we can
> currently do is to skip the test.
>
> I think we are overusing the Skipped machinery at the moment for platform
> differences. I think in many cases it would be better to instead land an
> expected failure result. On the other hand, one really great thing about the
> Skipped file is that there’s a complete list in the file, allowing everyone
> to see the list. It makes a good to do list, probably better than just a
> list of bugs. This made Darin Fisher’s recent “why are so many tests
> skipped, lets fix it” message possible.
>
>    -- Darin
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>