[webkit-dev] More on test flakiness

[Julie Parent]

As Eric just said to me in person, another option is to just re-run *any*
failing test twice, and only turn tree red if it fails twice.  (Chromium
just recently started doing this, and it has greatly improved our tree
greenness).  This obviously doesn't explicitly identify timing dependent
tests, but it solves the bigger issues that flaky tests cause.

On Wed, Dec 2, 2009 at 2:43 PM, Julie Parent
<jparent+webkit at gmail.com<jparent%2Bwebkit at gmail.com>
> wrote:

> In a recent code review where I was minimizing some test flakiness, Darin
> said:
> "I wish there was a way to isolate timing-dependent tests separately from
> the vast majority of tests that can run at any speed. I'd prefer to not have
> tests that pass or fail based on the speed or load of the computer, but if
> we do knowingly have them it would be *so* much better if they were
> identified somehow."
>
> I agree, and would like to implement something to address this.
>
> Possible ways to mark a test as timing-dependent:
>
>    - Put tests in a specific directory
>    - Append a suffix to the test name
>    - Add a function call to layoutTestController that is called explicitly
>    for timing dependent tests
>
> From here, the issue becomes how to use the knowledge.  Some ideas:
>
>    - If one of these tests fails, don't turn the bots red, turn some other
>    color
>    - If one of these tests fails, re-run it.  If it passes the second
>    time, consider it a normal pass
>    - Turn bots red as normal, but with an indicator that the test is known
>    timing dependent (if we used a suffix on the test name, I guess this would
>    just be obvious)
>
> Thoughts?
>
> Julie
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091202/7a15bdb0/attachment.html>