[webkit-dev] Skipping Flakey Tests
atwilson at google.com
Thu Oct 1 12:09:02 PDT 2009
OK, I agree as well - skipping is not a good solution here; I don't think
the status quo is perfect, yet probably not imperfect enough to do anything
I guess there's just a process wrinkle we need to address on the Chromium
side. It's easy to rebaseline a test in Chromium, but less easy to figure
out when it's safe to un-rebaseline it.
On Thu, Oct 1, 2009 at 11:57 AM, Eric Seidel <eseidel at google.com> wrote:
> I agree with Darin. I don't think that this is a good example of
> where skipping would be useful.
> I think more you're identifying that there is a test hierarchy problem
> here. Chromium really wants to base its tests off of some base "win"
> implementation, and then "win-apple", "win-chromium", "win-cairo"
> results could derive from that, similar to how "mac" and
> "mac-leopard", "mac-tiger", "mac-snowleopard" work.
> > I think we should skip only tests that endanger the testing strategy
> > they are super-slow, crash, or adversely affect other tests in some way.
> Back to the original topic: I do however see flakey tests as
> "endangering our testing strategy" because they provide false
> negatives, and greatly reduce the value of the layout tests and things
> which run the layout tests, like the buildbots or the commit-bot.
> I also agree with Darin's earlier comment that WebKit needs something
> like Chromium's multiple-expected results support so that we can
> continue to run flakey tests, even if they're flakey instead of having
> to resort to skipping them. But for now, skipping is the best we
> have, and I still encourage us to use it when necessary instead of
> leaving layout tests flakey. :)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev