[webkit-dev] EWS Comments on Bugzilla (Was: EWS now parses error logs in case of build failure)

Devin Rousso drousso at apple.com
Tue Nov 5 09:49:12 PST 2019 

Hi,

I may be alone here, but I actually found build/test(s) failure emails to be super useful.  The part that I did not find useful was that each time a build/test(s) would fail on a particular platform, there would be two comments/emails, which furthermore split the useful information into two pieces.  IIRC, the first comment/email would contain information about what build/test(s) failed, and the second comment/email was just an archive upload of the layout test results (if applicable).

I would find it very useful to get an email each time one of the bots had something go "wrong", and ideally include a link directly to the 'errors' log that Aakash mention in the beginning of this thread (or to the layout test results in the case of a test(s) failure).  Right now, if something goes "wrong", I have to be the one to discover it and then click through a few pages until I find the data that will actually help me solve the problem.

As a patch author, this is useful in letting me not have to remember to check my patch once a day in order to see if there are any red bubbles (something which I've forgotten to do, leaving a broken patch up for review thinking all was fine).  I can upload a patch and ideally leave it be, as I will be notified if the build/test(s) fails.

As a patch reviewer, I'm not sure this would be as useful, as I have a saved query that I use for finding all patches that need to be reviewed, and on a search result page there are no EWS bubbles anyways, so I can't see the info there.  I have to click into the bug regardless.  To Ryosuke's point, it may still be useful to get an email as a reviewer simply so I know that something failed and can potentially remember that fact before I click on a bug in the search results page, but I know personally that my mind doesn't really work that way :P

Thanks,
Devin

> On Nov 5, 2019, at 09:04, Alexey Proskuryakov <ap at webkit.org> wrote:
> 
> 
> 
>> 4 нояб. 2019 г., в 1:37 PM, Ryosuke Niwa <rniwa at webkit.org <mailto:rniwa at webkit.org>> написал(а):
>> 
>> 
>> On Mon, Nov 4, 2019 at 9:40 AM Alexey Proskuryakov <ap at webkit.org <mailto:ap at webkit.org>> wrote:
>> 
>> Can you elaborate on that, how exactly is e-mailing on first failure useful to reviewers?
>> 
>> Getting rid of Bugzilla comments was one of the goals of EWS rewrite, based on engineering feedback about noise in bugs and in e-mail, and I wholeheartedly agree with this feedback. So I think that comments are generally undesirable.
>> 
>> Since I don't understand what your precise scenario is, I may be make straw man arguments below, but here are some things that I think make the proposed behavior unhelpful (add a comment on first failure, or when all EWSes pass).
>> 
>> 1. EWS comments in Bugzilla are so annoying that some people take the radical step of manually hiding them. EWS history is archived anyway, there is no need to look into comments for it.
>> 
>> 2. There are often many people CC'ed on the bug to whom EWS data is irrelevant or even mysterious (e.g. reporters, web developers or non-reviewers). The noise is a slight annoyance, discouraging further participation in the project.
>> 
>> 3. I believe that for most reviewers, the mode of operation is one of the two: (1) do it when pinged directly, or (2) go over the review queue when one has the time. Getting EWS comments helps neither.
>> 
>> 4. Commenting when all EWSes pass is not very practical - it's too often that we have some stragglers that take days (or forever). I don't think that we can make it reliable even if we start actively policing EWS responsiveness.
>> 
>> 5. The reviewer likely wants to know the state of multiple EWSes if they are going to wait for EWS at all. What exactly are they going to do after getting an e-mail that one EWS failed?
>> 
>> I often use a EWS failure as a signal to wait reviewing a patch. Otherwise, a bug mail will stay in my inbox as one of items to get to.
>> 
>> I can see the usefulness in the somewhat unusual case of a super urgent patch. We may want multiple people to watch it, so that members of CC list would go and ask the patch author to update it with more urgency than e-mail allows for. I think that opt-in is a better mechanism for that, so that people who opted in would receive information about each EWS data point.
>> 
>> I think there is a value in knowing that a patch isn't ready instead of having to open the bug to realize that.
> 
> So just to clarify, 
> 
> - a major part of how you get to review bugs is by being CC'ed, and you review them when you have the time to read bugmail;
> - and you don't open the bug in Bugzilla if there is already an EWS failure by the time you read the e-mail where review is requested?
> 
> That's clearly a valid benefit. In my mind, it probably doesn't outweigh the downsides. On the other hand, yours is a voice of someone who reviews way more patches than Maciej and me combined these days, so maybe more e-mail is an overall benefit to many of the reviewers.
> 
> - Alexey
> 
> 
> 
>> - R. Niwa
>>> 3 нояб. 2019 г., в 6:58 PM, Maciej Stachowiak <mjs at apple.com <mailto:mjs at apple.com>> написал(а):
>>> 
>>> 
>>> I think they are useful to actual and potential reviewers. Direct email to the patch author is not something anyone can Cc themselves on, and is not archived, so seems like a strictly worse form of communication.
>>> 
>>>> On Nov 2, 2019, at 9:34 AM, Alexey Proskuryakov <ap at apple.com <mailto:ap at apple.com>> wrote:
>>>> 
>>>> 
>>>> My preference is still e-mailing the patch author directly (possibly, also having an option to opt in for anyone). Bugzilla comments will always be irrelevant for most people CC'ed on the bug, and they are almost always undesirable to keep within the discussion flow.
>>>> 
>>>> - Alexey
>>>> 
>>>>> 1 нояб. 2019 г., в 18:28, Aakash Jain <aakash_jain at apple.com <mailto:aakash_jain at apple.com>> написал(а):
>>>>> 
>>>>> Sounds good. I prefer the single comment when the first failure occur. That way notification would be sent as soon as the first failure happens.
>>>>> 
>>>>> I'll implement that (assuming it's acceptable to everyone).
>>>>> 
>>>>> Thanks
>>>>> Aakash
>>>>> 
>>>>>> On Nov 1, 2019, at 8:35 PM, Maciej Stachowiak <mjs at apple.com <mailto:mjs at apple.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> How about only a single comment when the first failure occurs? (Or else when all bots pass, if there is never a failure.)
>>>>>> 
>>>>>> This should help the author, the reviewer, and anyone else cc’d, without being too spammy.
>>>>>> 
>>>>>>> On Nov 1, 2019, at 5:20 PM, Aakash Jain <aakash_jain at apple.com <mailto:aakash_jain at apple.com>> wrote:
>>>>>>> 
>>>>>>> Hi Ryosuke,
>>>>>>> 
>>>>>>> Many people didn't like the noise by the EWS comments, and we removed the comments as per previous discussion in: https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html <https://lists.webkit.org/pipermail/webkit-dev/2019-June/030683.html>.
>>>>>>> 
>>>>>>> I agree with your point that having some kind of notification might be useful.
>>>>>>> 
>>>>>>> I proposed some ideas in https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html <https://lists.webkit.org/pipermail/webkit-dev/2019-September/030798.html>, but didn't get much feedback. If we can all agree on a solution, I can look into implementing it.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Aakash
>>>>>>> 
>>>>>>>> On Oct 30, 2019, at 1:03 AM,
>>>>>>>> - R. Niwa
>>>>>>>> <rniwa at webkit.org <mailto:rniwa at webkit.org>> wrote:
>>>>>>>> 
>>>>>>>> These enhancements are great. There is one feature of the old EWS that I really miss, which is that I used to get emails when some EWS failed. With new EWS, I have to keep checking back the bugzilla to see if any of them have failed periodically.
>>>>>>>> 
>>>>>>>> Can we add a feature to opt into such an email notification? Maybe a flag on a patch or JSON configuration file somewhere.
>>>>>>>> 
>>>>>>>> - R. Niwa
>>>>>>>> 
>>>>>>>> On Tue, Oct 29, 2019 at 4:05 PM Aakash Jain <aakash_jain at apple.com <mailto:aakash_jain at apple.com>> wrote:
>>>>>>>> Hi Everyone,
>>>>>>>> 
>>>>>>>> I am happy to announce another EWS feature.
>>>>>>>> 
>>>>>>>> From now on, in case of build failure, EWS will parse the errors and display them in a separate 'errors' log. You wouldn't have to search through thousands of lines of logs to find the error message.
>>>>>>>> 
>>>>>>>> For example, in https://ews-build.webkit.org/#/builders/16/builds/6054 <https://ews-build.webkit.org/#/builders/16/builds/6054>, in step #7 WebKit failed to compile. Complete logs (stdio) are 38,000+ lines, and the error is not at the end of the logs. Normally, it requires some searching through the logs to find the relevant errors. But now, there is another 'errors' log, which contains just the relevant 11 lines (containing error and few related lines to provide additional context).
>>>>>>>> 
>>>>>>>> Hopefully this would save some time and efforts previously spent on searching through the large logs.
>>>>>>>> 
>>>>>>>> Note that this information is not displayed in status-bubble tool-tip, since this might be lot of text to display in the tooltip. My further plan is to make this information more readily available, by adding it to a custom designed page which will open on clicking the status bubble https://webkit.org/b/197522 <https://webkit.org/b/197522>
>>>>>>>> 
>>>>>>>> Please let me know if you notice any issues or have any feedback.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Aakash
>>>>>>>> 
>>>>>>>> Reference: https://webkit.org/b/203418 <https://webkit.org/b/203418>
>>>>>>>> _______________________________________________
>>>>>>>> webkit-dev mailing list
>>>>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>>>>>> -- 
>>>>>>>> - R. Niwa
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> webkit-dev mailing list
>>>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> webkit-dev mailing list
>>>>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
>>>> 
>>>> - Alexey
>>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
>> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> 
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20191105/35aec1a8/attachment.htm>

More information about the webkit-dev mailing list