[webkit-dev] More on test flakiness
mjs at apple.com
Fri Dec 4 15:48:52 PST 2009
On Dec 2, 2009, at 2:43 PM, Julie Parent wrote:
> In a recent code review where I was minimizing some test flakiness,
> Darin said:
> "I wish there was a way to isolate timing-dependent tests separately
> from the vast majority of tests that can run at any speed. I'd
> prefer to not have tests that pass or fail based on the speed or
> load of the computer, but if we do knowingly have them it would be
> *so* much better if they were identified somehow."
> I agree, and would like to implement something to address this.
> Possible ways to mark a test as timing-dependent:
> Put tests in a specific directory
> Append a suffix to the test name
> Add a function call to layoutTestController that is called
> explicitly for timing dependent tests
Out of these ways to mark timing-sensitive tests, I prefer a specific
directory. This is a list we should try to drive to 0 over time, so
having a dedicated directory would make it easiest to see how far off
we are from that goal.
> From here, the issue becomes how to use the knowledge. Some ideas:
> If one of these tests fails, don't turn the bots red, turn some
> other color
> If one of these tests fails, re-run it. If it passes the second
> time, consider it a normal pass
> Turn bots red as normal, but with an indicator that the test is
> known timing dependent (if we used a suffix on the test name, I
> guess this would just be obvious)
I prefer the following:
- If one of the tests fails a first time, re-run it a second time.
- If it fails the second time, mark the bot red (with an indication
that the test is timing sensitive).
- If the test failed the first time only, mark the bot some color
other than red or green, so we are reminded that this specific test is
flaking out without actually marking the build as broken.
Some theoretically timing-dependent tests may turn out to fail
exceedingly often, or may turn out not to really be timing-dependent
in practice, or may become more flaky at some future point, so it
would be good to have some indicator of that in the buildbot.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev