[webkit-dev] Closing the loop on flaky tests (was Re: Flaky test hit list)

[Adam Barth]

On Tue, Oct 19, 2010 at 11:44 AM, Alexey Proskuryakov <ap at webkit.org> wrote:
> 19.10.2010, в 11:16, Adam Barth написал(а):
>> Also, these bugs are close to the end of their
>> lifecycle (because their patch is about to land), so they shouldn't
>> generate more than 3 or 4 emails each.  That boils down to about one
>> or two emails per week for the flakiest test.
>
> One e-mail (per week?) would perhaps make sense, even though "flaky test" is sometimes "flaky code", so the blame becomes misplaced. Getting 3-4 automated e-mails per bug seems over the board.

Maybe the thing to do is CC the author of the flaky test for the one
bug comment and then immediately unCC them.  That way they don't see
the rest of the traffic on the bug.

Adam


> I agree that raising awareness of which tests or code areas are flaky seems useful. One problem I personally had was with digging up data on flakiness. The link for a dashboard that I found was <http://test-results.appspot.com/dashboards/flakiness_dashboard.html> - the URL was freezing my browser for several minutes on each move, and I couldn't make sense of what it was telling me UI-wise quickly enough. I'm not even sure how it's related to flakiness seen by commit queue, as it seems to be about chromium.
>
> Is there a better data source that I missed?
>
> - WBR, Alexey Proskuryakov
>
>