[webkit-dev] Skipping Flakey Tests

[Eric Seidel]

I agree with Darin.  I don't think that this is a good example of
where skipping would be useful.

I think more you're identifying that there is a test hierarchy problem
here.  Chromium really wants to base its tests off of some base "win"
implementation, and then "win-apple", "win-chromium", "win-cairo"
results could derive from that, similar to how "mac" and
"mac-leopard", "mac-tiger", "mac-snowleopard" work.

> I think we should skip only tests that endanger the testing strategy because
> they are super-slow, crash, or adversely affect other tests in some way.

Back to the original topic:  I do however see flakey tests as
"endangering our testing strategy" because they provide false
negatives, and greatly reduce the value of the layout tests and things
which run the layout tests, like the buildbots or the commit-bot.

I also agree with Darin's earlier comment that WebKit needs something
like Chromium's multiple-expected results support so that we can
continue to run flakey tests, even if they're flakey instead of having
to resort to skipping them.  But for now, skipping is the best we
have, and I still encourage us to use it when necessary instead of
leaving layout tests flakey. :)

-eric

On Thu, Oct 1, 2009 at 11:47 AM, Darin Adler <darin at apple.com> wrote:
> On Oct 1, 2009, at 11:41 AM, Drew Wilson wrote:
>
>> I don't have an opinion about flakey tests, but flat-out-busted tests
>> should get skipped. Any thoughts/objections?
>
> I object.
>
> If a test fails on some platforms and succeeds on others, we should have the
> success result checked in as the default case, and the failure as an
> exception. And we should structure test results and exceptions so that it’s
> easy to get the expected failure on the right platforms and success on
> others. Your story about a slight inconvenience because a test failed on the
> base Windows WebKit but succeeded on the Chromium WebKit does not seem like
> a reason to change this!
>
> Skipping the test does not seem like a good thing to do for the long term
> health of the project. It is good to exercise all the other code each test
> covers and also to notice when a test result gets even worse or gets better
> when a seemingly unrelated change is made.
>
> I think we should skip only tests that endanger the testing strategy because
> they are super-slow, crash, or adversely affect other tests in some way.
>
>    -- Darin
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>