[webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)

Mon Dec 2 17:19:29 PST 2019

There were multiple ideas discussed in this thread. I would like to gather more data about what do most people prefer. I have sent out a short survey in https://lists.webkit.org/pipermail/webkit-dev/2019-December/030980.html <https://lists.webkit.org/pipermail/webkit-dev/2019-December/030980.html>

Thanks
Aakash

> On Nov 5, 2019, at 12:04 PM, Alexey Proskuryakov <ap at webkit.org> wrote:
> 
> 
> 
>> 4 нояб. 2019 г., в 1:37 PM, Ryosuke Niwa <rniwa at webkit.org <mailto:rniwa at webkit.org>> написал(а):
>> 
>> 
>> On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov <ap at webkit.org <mailto:ap at webkit.org>> wrote:
>> 
>> Can you elaborate on that, how exactly is e-mailing on first failure useful to reviewers?
>> 
>> Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based on engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree with this feedback. So I think that comments are generally undesirable.
>> 
>> Since I don't understand what your precise scenario is, I may be make straw man arguments below, but here are some things that I think make the proposed behavior unhelpful (add a comment on first failure, or when all EWSes pass).
>> 
>> 1. EWS comments in Bugzilla are so annoying that some people take the radical step of manually hiding them. EWS history is archived anyway, there is no need to look into comments for it.
>> 
>> 2. There are often many people CC'ed on the bug to whom EWS data is irrelevant or even mysterious (e.g. reporters, web developers or non-reviewers). The noise is a slight annoyance, discouraging further participation in the project.
>> 
>> 3. I believe that for most reviewers, the mode of operation is one of the two: (1) do it when pinged directly, or (2) go over the review queue when one has the time. Getting EWS comments helps neither.
>> 
>> 4. Commenting when all EWSes pass is not very practical - it's too often that we have some stragglers that take days (or forever). I don't think that we can make it reliable even if we start actively policing EWS responsiveness.
>> 
>> 5. The reviewer likely wants to know the state of multiple EWSes if they are going to wait for EWS at all. What exactly are they going to do after getting an e-mail that one EWS failed?
>> 
>> I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, a bug mail will stay in my inbox as one of items to get to.
>> 
>> I can see the usefulness in the somewhat unusual case of a super urgent patch. We may want multiple people to watch it, so that members of CC list would go and ask the patch author to update it with more urgency than e-mail allows for. I think that opt-in is a better mechanism for that, so that people who opted in would receive information about each EWS data point.
>> 
>> I think there is a value in knowing that a patch isn't ready instead of having to open the bug to realize that.
> 
> So just to clarify, 
> 
> - a major part of how you get to review bugs is by being CC'ed, and you review them when you have the time to read bugmail;
> - and you don't open the bug in Bugzilla if there is already an EWS failure by the time you read the e-mail where review is requested?
> 
> That's clearly a valid benefit. In my mind, it probably doesn't outweigh the downsides. On the other hand, yours is a voice of someone who reviews way more patches than Maciej and me combined these days, so maybe more e-mail is an overall benefit to many of the reviewers.
> 
> - Alexey
> 
> 
> 
>> - R. Niwa
>>> 3 нояб. 2019 г., в 6:58 PM, Maciej Stachowiak <mjs at apple.com <mailto:mjs at apple.com>> написал(а):
>>> 
>>> 
>>> I think they are useful to actual and potential reviewers. Direct email to the patch author is not something anyone can Cc themselves on, and is not archived, so seems like a strictly worse form of communication.
>>> 
>>>> On Nov 2, 2019, at 9:34 AM, Alexey Proskuryakov <ap at apple.com <mailto:ap at apple.com>> wrote:
>>>> 
>>>> 
>>>> My preference is still e-mailing the patch author directly (possibly, also having an option to opt in for anyone). Bugzilla comments will always be irrelevant for most people CC'ed on the bug, and they are almost always undesirable to keep within the discussion flow.
>>>> 
>>>> - Alexey
>>>> 
>>>>> 1 нояб. 2019 г., в 18:28, Aakash Jain <aakash_jain at apple.com <mailto:aakash_jain at apple.com>> написал(а):
>>>>> 
>>>>> Sounds good. I prefer the single comment when the first failure occur. That way notification would be sent as soon as the first failure happens.
>>>>> 
>>>>> I'll implement that (assuming it's acceptable to everyone).
>>>>> 
>>>>> Thanks
>>>>> Aakash
>>>>> 
>>>>>> On Nov 1, 2019, at 8:35 PM, Maciej Stachowiak <mjs at apple.com <mailto:mjs at apple.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> How about only a single comment when the first failure occurs? (Or else when all bots pass, if there is never a failure.)
>>>>>> 
>>>>>> This should help the author, the reviewer, and anyone else cc’d, without being too spammy.
>>>>>> 
>>>>>>> On Nov 1, 2019, at 5:20 PM, Aakash Jain <aakash_jain at apple.com <mailto:aakash_jain at apple.com>> wrote:
>>>>>>> 
>>>>>>> Hi Ryosuke,
>>>>>>> 
>>>>>>> Many people didn't like the noise by the EWS comments, and we removed the comments as per previous discussion in: https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html <https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html>.
>>>>>>> 
>>>>>>> I agree with your point that having some kind of notification might be useful.
>>>>>>> 
>>>>>>> I proposed some ideas in https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html <https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html>, but didn't get much feedback. If we can all agree on a solution, I can look into implementing it.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Aakash
>>>>>>> 
>>>>>>>> On Oct 30, 2019, at 1:03 AM,
>>>>>>>> - R. Niwa
>>>>>>>> <rniwa at webkit.org <mailto:rniwa at webkit.org>> wrote:
>>>>>>>> 
>>>>>>>> These enhancements are great. There is one feature of the old EWS that I really miss, which is that I used to get emails when some EWS failed. With new EWS, I have to keep checking back the bugzilla to see if any of them have failed periodically.
>>>>>>>> 
>>>>>>>> Can we add a feature to opt into such an email notification? Maybe a flag on a patch or JSON configuration file somewhere.
>>>>>>>> 
>>>>>>>> - R. Niwa
>>>>>>>> 
>>>>>>>> On Tue, Oct 29, 2019 at 4:05 PM Aakash Jain <aakash_jain at apple.com <mailto:aakash_jain at apple.com>> wrote:
>>>>>>>> Hi Everyone,
>>>>>>>> 
>>>>>>>> I am happy to announce another EWS feature.
>>>>>>>> 
>>>>>>>> From now on, in case of build failure, EWS will parse the errors and display them in a separate 'errors' log. You wouldn't have to search through thousands of lines of logs to find the error message.
>>>>>>>> 
>>>>>>>> For example, in https://ews-build.webkit.org/#/builders/16/builds/6054 <https://ews-build.webkit.org/#/builders/16/builds/6054>, in step #7 WebKit failed to compile. Complete logs (stdio) are 38,000+ lines, and the error is not at the end of the logs. Normally, it requires some searching through the logs to find the relevant errors. But now, there is another 'errors' log, which contains just the relevant 11 lines (containing error and few related lines to provide additional context).
>>>>>>>> 
>>>>>>>> Hopefully this would save some time and efforts previously spent on searching through the large logs.
>>>>>>>> 
>>>>>>>> Note that this information is not displayed in status-bubble tool-tip, since this might be lot of text to display in the tooltip. My further plan is to make this information more readily available, by adding it to a custom designed page which will open on clicking the status bubble https://webkit.org/b/197522 <https://webkit.org/b/197522>
>>>>>>>> 
>>>>>>>> Please let me know if you notice any issues or have any feedback.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Aakash
>>>>>>>> 
>>>>>>>> Reference: https://webkit.org/b/203418 <https://webkit.org/b/203418>
>>>>>>>> _______________________________________________
>>>>>>>> webkit-dev mailing list
>>>>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>>>>>> -- 
>>>>>>>> - R. Niwa
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> webkit-dev mailing list
>>>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> webkit-dev mailing list
>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>> 
>>>> - Alexey
>>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> 
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20191202/9f578098/attachment.htm>