[webkit-dev] Skipping Flakey Tests

[Drew Wilson]

OK, I agree as well - skipping is not a good solution here; I don't think
the status quo is perfect, yet probably not imperfect enough to do anything
about :)
I guess there's just a process wrinkle we need to address on the Chromium
side. It's easy to rebaseline a test in Chromium, but less easy to figure
out when it's safe to un-rebaseline it.
-atw

On Thu, Oct 1, 2009 at 11:57 AM, Eric Seidel <eseidel at google.com> wrote:

> I agree with Darin.  I don't think that this is a good example of
> where skipping would be useful.
>
> I think more you're identifying that there is a test hierarchy problem
> here.  Chromium really wants to base its tests off of some base "win"
> implementation, and then "win-apple", "win-chromium", "win-cairo"
> results could derive from that, similar to how "mac" and
> "mac-leopard", "mac-tiger", "mac-snowleopard" work.
>
> > I think we should skip only tests that endanger the testing strategy
> because
> > they are super-slow, crash, or adversely affect other tests in some way.
>
> Back to the original topic:  I do however see flakey tests as
> "endangering our testing strategy" because they provide false
> negatives, and greatly reduce the value of the layout tests and things
> which run the layout tests, like the buildbots or the commit-bot.
>
> I also agree with Darin's earlier comment that WebKit needs something
> like Chromium's multiple-expected results support so that we can
> continue to run flakey tests, even if they're flakey instead of having
> to resort to skipping them.  But for now, skipping is the best we
> have, and I still encourage us to use it when necessary instead of
> leaving layout tests flakey. :)
>
> -eric
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091001/e25da0b4/attachment.html>