[webkit-dev] Closing the loop on flaky tests (was Re: Flaky test hit list)

[Eric Seidel]

Sorry, wrong account.

On Tue, Oct 19, 2010 at 1:59 PM, Eric Seidel <eseidel at google.com> wrote:
> On Tue, Oct 19, 2010 at 1:45 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>> It looks like the bot is adding a comment to the bug with the patch that was being processed when flakiness was detected, not the one that originally landed the tests believed to be flaky. Is that right?
>
> Correct.  The original message was intended as a notice to the person
> who's patch it was, explaining why there patch was taking so long.
> (Flaky tests often double, triple or more the total time it takes to
> commit a patch.)
>
>> If so, that doesn't seem like a great way to notify the author of the original test. It seems like it would be better to comment in the bug that added the test.
>
> Interesting possibility.
>
> What started this discussion is that last night we made the
> commit-queue CC the original author of the flaky test every time we
> posted one of these "we're slow to commit your patch because these
> tests are flaky" messages.  4 flakes tests were hit last night after
> we added that message, 3 of which were caused by tests authored by
> Alexey -- hence he had extra mail in his inbox this morning and this
> discussion ensued.  As Adam noted, this was likely a statistical
> fluke.
>
> Commenting on the original bug is a good idea, assuming the original
> commit had a bug link.
>
>>  To be fair, it's also possible that the new patch caused the flakiness, so a separate comment there could be useful. Perhaps it would be useful to determine if the test in question has a track record of flakiness. If not, then maybe the presumption should be that the patch is the problem, not the test. On the other hand, if the test has always been flaky, then the new patch probably has nothing to do with it.
>
> Definitely possible, but I've not ever seen this happen in practice.
> Generally either the commit-queue fails due to the new flakiness, or
> it gets landed and someone later finds and removes it.  It would be
> rare to have the new patch be adding new flakiness and the old test
> author getting CC'd.  Actually in that case, these CC's seem more
> useful, as the old test author would be made aware of changes causing
> his/her old test to go flaky.
>
> -eric
>