[webkit-dev] More on test flakiness

[Julie Parent]

In a recent code review where I was minimizing some test flakiness, Darin
said:
"I wish there was a way to isolate timing-dependent tests separately from
the vast majority of tests that can run at any speed. I'd prefer to not have
tests that pass or fail based on the speed or load of the computer, but if
we do knowingly have them it would be *so* much better if they were
identified somehow."

I agree, and would like to implement something to address this.

Possible ways to mark a test as timing-dependent:

   - Put tests in a specific directory
   - Append a suffix to the test name
   - Add a function call to layoutTestController that is called explicitly
   for timing dependent tests

>From here, the issue becomes how to use the knowledge.  Some ideas:

   - If one of these tests fails, don't turn the bots red, turn some other
   color
   - If one of these tests fails, re-run it.  If it passes the second time,
   consider it a normal pass
   - Turn bots red as normal, but with an indicator that the test is known
   timing dependent (if we used a suffix on the test name, I guess this would
   just be obvious)

Thoughts?

Julie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091202/b0da674a/attachment.html>