[webkit-dev] More on test flakiness

[Maciej Stachowiak]

On Dec 2, 2009, at 2:43 PM, Julie Parent wrote:

> In a recent code review where I was minimizing some test flakiness,  
> Darin said:
> "I wish there was a way to isolate timing-dependent tests separately  
> from the vast majority of tests that can run at any speed. I'd  
> prefer to not have tests that pass or fail based on the speed or  
> load of the computer, but if we do knowingly have them it would be  
> *so* much better if they were identified somehow."
>
> I agree, and would like to implement something to address this.
>
> Possible ways to mark a test as timing-dependent:
> Put tests in a specific directory
> Append a suffix to the test name
> Add a function call to layoutTestController that is called  
> explicitly for timing dependent tests
Out of these ways to mark timing-sensitive tests, I prefer a specific  
directory. This is a list we should try to drive to 0 over time, so  
having a dedicated directory would make it easiest to see how far off  
we are from that goal.


> From here, the issue becomes how to use the knowledge.  Some ideas:
> If one of these tests fails, don't turn the bots red, turn some  
> other color
> If one of these tests fails, re-run it.  If it passes the second  
> time, consider it a normal pass
> Turn bots red as normal, but with an indicator that the test is  
> known timing dependent (if we used a suffix on the test name, I  
> guess this would just be obvious)
> Thoughts?

I prefer the following:

- If one of the tests fails a first time, re-run it a second time.
- If it fails the second time, mark the bot red (with an indication  
that the test is timing sensitive).
- If the test failed the first time only, mark the bot some color  
other than red or green, so we are reminded that this specific test is  
flaking out without actually marking the build as broken.

Some theoretically timing-dependent tests may turn out to fail  
exceedingly often, or may turn out not to really be timing-dependent  
in practice, or may become more flaky at some future point, so it  
would be good to have some indicator of that in the buildbot.

Regards,
Maciej

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091204/7727b588/attachment.html>