[webkit-dev] Requesting feedback about EWS comments on Bugzilla bugs

[Michael Catanzaro]

1-4 all seem uncontroversial, especially with Tim's suggested 
improvement to immediately leave a "some queues failed" comment.

On Sat, Jun 15, 2019 at 11:13 PM, Aakash Jain <aakash_jain at apple.com> 
wrote:
> 5) Do not comment on bugzilla bug at all, instead send email to the 
> author of the patch.
> Pros: less noisy, also this will allow to include more detailed 
> information about the failure in email.
> Cons: reviewers would have to click status-bubbles to see the 
> failures, failure information is not immediately present in the 
> comments.

Well I think this would be OK, but next we'll be complaining about the 
emails. The underlying problem is excessive test flakiness. We're just 
not doing a very good job of tracking flaky tests and EWS isn't good at 
detecting flaky tests automatically. (Often a flaky test will fail 
twice in the same run, and that gets detected as a test failure.) 
Instead of reporting bugs for such tests, we're ignoring them because 
there are so many and we're so used to it.

commit-queue is able to report bugs when it detects a flaky patch, but 
EWS doesn't do this. That might be a positive change. Of course, EWS 
would need to be smart enough to not report a bug if the flakiness is 
triggered by the patch itself, which might not be simple to determine.

I don't have any concrete suggestions here. Just brainstorming.