[webkit-dev] Closing the loop on flaky tests (was Re: Flaky test hit list)

[Adam Barth]

On Tue, Oct 19, 2010 at 1:45 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> On Oct 19, 2010, at 1:30 PM, Adam Barth wrote:
>> On Tue, Oct 19, 2010 at 12:41 PM, Alexey Proskuryakov <ap at webkit.org> wrote:
>>> 19.10.2010, в 12:33, Adam Barth написал(а):
>>>> Maybe the thing to do is CC the author of the flaky test for the one
>>>> bug comment and then immediately unCC them.  That way they don't see
>>>> the rest of the traffic on the bug.
>>>
>>> That would still be two e-mails about a bug the person otherwise doesn't want to know about. I don't think that CC'ing is the right approach.
>>
>> Do you see changes to bugs when you get removed from the CC?  Do you
>> have another suggestion for how to providing feedback to authors of
>> flaky tests?
>
> It looks like the bot is adding a comment to the bug with the patch that was being processed when flakiness was detected, not the one that originally landed the tests believed to be flaky. Is that right? If so, that doesn't seem like a great way to notify the author of the original test. It seems like it would be better to comment in the bug that added the test. To be fair, it's also possible that the new patch caused the flakiness, so a separate comment there could be useful. Perhaps it would be useful to determine if the test in question has a track record of flakiness. If not, then maybe the presumption should be that the patch is the problem, not the test. On the other hand, if the test has always been flaky, then the new patch probably has nothing to do with it.

Another option is to file a new bug about the flakiness and ping that
bug when we observe the test flake out.

Adam