In Chromium-land, we have been working to eliminate flaky layout tests.  One thing that has helped us gain insight into flaky tests is the <a href="http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html">flakiness dashboard</a>.  I&#39;d love to see this available for <a href="http://webkit.org">webkit.org</a> as well.  I&#39;m happy to do the work to set up the dashboard itself, but one thing we&#39;d need is for run-webkit-tests to output an extra json file, and for the build bots to store this file.<div>
<br></div><div>Is a flakiness dashboard something you&#39;d be interested in?  Is storing the extra data file ok (the largest <a href="http://chromium.org">chromium.org</a> one is &lt; 1MB and the <a href="http://webkit.org">webkit.org</a> ones should be much smaller since it is based on # failing tests)? Anyone who knows Perl interested in adding the support to run-webkit-tests to output the json (we can provide details on the format)?<br>
<div><div><br></div><div>Julie</div><div><br></div><div><div>Q: Why do we care about flaky layout tests?</div><div>A: They cost us time.  Specfically, they cause the commit-queue to reject good patches, lead engineers to spend extra time spent determining if they caused the tree to go red, mask real regressions because the tree was already red, put an extra burden on ports trying to determine if their port is bad or the test is just bad, etc.</div>
<div><br></div><div>Q: How can I help fix flaky tests/not introduce more flaky tests?</div><div>A: One big, easily fixable, source of flakiness is setTimeouts.  Whenever possible, use specific events rather than relying on setTimeouts.  Don&#39;t use setTimeout to wait for resources to load.  Use onload events instead (iframe, image, body all have onload events).  If there are no events possible, and you need time to elapse before checking something, provide a little extra wiggle room.</div>
<div><br></div><div>Q: Do you have examples of these sorts of easy fixes?</div><div>A: Timeout increase:<span class="Apple-style-span" style="font-family: arial, sans-serif; font-size: 13px; border-collapse: collapse; "> <a href="http://trac.webkit.org/changeset/51150" target="_blank" style="color: rgb(7, 77, 143); ">http://trac.webkit.org/changeset/51150</a>. Remove extra watchdog setTimeout: <a href="http://trac.webkit.org/changeset/51120" target="_blank" style="color: rgb(7, 77, 143); ">http://trac.webkit.org/changeset/51120</a>. Remove unnecessary setTimeout: <a href="http://trac.webkit.org/changeset/51088" target="_blank" style="color: rgb(7, 77, 143); ">http://trac.webkit.org/changeset/51088</a>. Use onload to detect iframe loading: <a href="http://trac.webkit.org/changeset/49592" target="_blank" style="color: rgb(7, 77, 143); ">http://trac.webkit.org/changeset/49592</a></span></div>
<div><br></div><div>Q. I am still reading, can I see a pretty picture?<br>A: Sure. Here is the dashboard results clearly showing 2 tests that got fixed.  You can see the flakiness before (to the right) and the new non-flakiness after (to the left).  <a href="http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=LayoutTests%2Ftransitions%2Fzero-duration-with-non-zero-delay-end.html%20LayoutTests%2Ffast%2Fencoding%2Fchar-encoding.html">http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=LayoutTests%2Ftransitions%2Fzero-duration-with-non-zero-delay-end.html%20LayoutTests%2Ffast%2Fencoding%2Fchar-encoding.html</a></div>
</div>
</div></div>