[Webkit-unassigned] [Bug 61061] TestFailures page blames arbitrary revisions for breaking flaky tests

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=61061





--- Comment #6 from Adam Roben (:aroben) <aroben at apple.com>  2011-06-29 13:09:24 PST ---
(In reply to comment #5)
> View in context: https://bugs.webkit.org/attachment.cgi?id=99100&action=review
> 
> > Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/FlakyLayoutTestDetector.js:35
> > +            return false;
> 
> Since the expected output of this function is an Array, it might make more sense to return null (still falsey).
> 
> > Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:108
> > +            var newFlakyTests = self._flakinessDetector.incorporateTestResults(nextBuildName, tests, tooManyFailures);
> > +            if (newFlakyTests.length) {
> 
> If there were tooManyFailures, then this would return "false" (or maybe "null") in the future and the .length check would through an error. This should probably become:
> 
>     if (newFlakyTests && newFlakyTests.length) { ... }

I think I'll change to returning an empty array in this case. The false is leftover from an earlier version of this function that just returned a boolean.

> > Tools/ChangeLog:31
> > +        We tell our caller to keep calling until all current failures have been explained and we've
> > +        gone through 5 builds without any new flaky tests being identified.
> 
> How far back does this normally go? Just 5-10 builds? 100 builds? Is there a limit on how far back this looks or could it go back to the start (if things were flaky).

It could go all the way back to the start. In practice it only goes back fewer than 20 builds in most cases.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.