[webkit-dev] Reducing layout test flakiness

[Maciej Stachowiak]

On Nov 20, 2009, at 2:49 PM, Julie Parent wrote:

> In Chromium-land, we have been working to eliminate flaky layout  
> tests.  One thing that has helped us gain insight into flaky tests  
> is the flakiness dashboard.  I'd love to see this available for  
> webkit.org as well.  I'm happy to do the work to set up the  
> dashboard itself, but one thing we'd need is for run-webkit-tests to  
> output an extra json file, and for the build bots to store this file.
>
> Is a flakiness dashboard something you'd be interested in?  Is  
> storing the extra data file ok (the largest chromium.org one is <  
> 1MB and the webkit.org ones should be much smaller since it is based  
> on # failing tests)? Anyone who knows Perl interested in adding the  
> support to run-webkit-tests to output the json (we can provide  
> details on the format)?

Sounds like a good plan to me.

>
> Julie
>
> Q: Why do we care about flaky layout tests?
> A: They cost us time.  Specfically, they cause the commit-queue to  
> reject good patches, lead engineers to spend extra time spent  
> determining if they caused the tree to go red, mask real regressions  
> because the tree was already red, put an extra burden on ports  
> trying to determine if their port is bad or the test is just bad, etc.
>
> Q: How can I help fix flaky tests/not introduce more flaky tests?
> A: One big, easily fixable, source of flakiness is setTimeouts.   
> Whenever possible, use specific events rather than relying on  
> setTimeouts.  Don't use setTimeout to wait for resources to load.   
> Use onload events instead (iframe, image, body all have onload  
> events).  If there are no events possible, and you need time to  
> elapse before checking something, provide a little extra wiggle room.

This is something we should watch for when writing new tests and  
reviewing patches that include tests. If the test case uses setTimeout  
or setInterval, that should be a huge red flag.

>
> Q: Do you have examples of these sorts of easy fixes?
> A: Timeout increase: http://trac.webkit.org/changeset/51150. Remove  
> extra watchdog setTimeout: http://trac.webkit.org/changeset/51120.  
> Remove unnecessary setTimeout: http://trac.webkit.org/changeset/ 
> 51088. Use onload to detect iframe loading: http://trac.webkit.org/changeset/49592
>
> Q. I am still reading, can I see a pretty picture?
> A: Sure. Here is the dashboard results clearly showing 2 tests that  
> got fixed.  You can see the flakiness before (to the right) and the  
> new non-flakiness after (to the left).  http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests 
> =LayoutTests%2Ftransitions%2Fzero-duration-with-non-zero-delay- 
> end.html%20LayoutTests%2Ffast%2Fencoding%2Fchar-encoding.html
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20091121/82ef386d/attachment.html>