[Webkit-unassigned] [Bug 29939] http/tests/xmlhttprequest/basic-auth.html timed out on Leopard bot

bugzilla-daemon at webkit.org bugzilla-daemon at webkit.org
Fri Jan 22 12:15:11 PST 2010


https://bugs.webkit.org/show_bug.cgi?id=29939





--- Comment #10 from Eric Seidel <eric at webkit.org>  2010-01-22 12:15:10 PST ---
I agree that a test expectations mechanism is the right way to go.  Sadly we
don't have such yet.  Leaving the bots red until some future time when we do is
silly.

I also think it's silly for one test to hold 11,000 tests hostage.  Which is
exactly what flakey tests do.  They reduce the usefulness of the rest of the
tests, but making it harder for someone to know if their patch is correct or
not.  People stop trusting the tests and bots.

By saying "but what if it catches a crasher later" you're arguing that we
should exchange some current time value for some unlikely future time value. 
The current value is the value of green bots catching real regressions quickly
because people know that red means they broken stuff.  The unlikely future
value is that this test would be the only one to catch some crasher.  We could
put numbers on such a value estimate, but I assure you the future value is not
worth the current cost.

I agree skipping tests is less than ideal, but it's the only tool we currently
have to keep the bots green.  That or rollouts.  Since we can't rollout the
change that broke this test, or realistically the one which added it, the best
solution we have is to skip it with plans to mark it flakey instead when we
have the technology. :)

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.



More information about the webkit-unassigned mailing list