Planned EWS improvements
Hi Everyone, I have been making number of improvements to EWS. I also have various planned improvements to EWS. I wanted to reach out to you guys to see if anyone wants me to prioritize any particular improvement(s). If there is any improvement which you want to see and is not listed below, please feel free to let me know. Also most of the queues have been transitioned from old to new EWS and I am working on the remaining ones (jsc, windows and commit-queue). Here is the list of improvements (in no particular order): 1) Develop a webpage showing summary of EWS builds for a patch. This page would provide the summary of important build-steps, high-level details about the failure (e.g.: name of the tests which failed, or possibly relevant build failure logs), and include link(s) to the Buildbot page(s). This page will open on clicking the status-bubbles (and would be replacement of old EWS status page like https://webkit-queues.webkit.org/patch/379563/win-ews <https://webkit-queues.webkit.org/patch/379563/win-ews>). Currently clicking the status-bubble opens the Buildbot build page, which contains a lot of infrastructure details, and probably is information-overload for many engineers, so this summary page should help with that. https://webkit.org/b/197522 <https://webkit.org/b/197522> 2) Redesign status-bubble tooltip to include more detailed information about failures (e.g.: each test failure name along-with url to flakiness dashboard, and url to complete results.html file, as suggested by David Kilzer in https://lists.webkit.org/pipermail/webkit-dev/2019-September/030799.html <https://lists.webkit.org/pipermail/webkit-dev/2019-September/030799.html>). We should also add the tooltip support for iPad/iPhone https://webkit.org/b/201940 <https://webkit.org/b/201940> 3) Add a way to retry a patch in EWS. This would allow engineers to confirm that the failures indicated by EWS aren't flaky/incorrect. Maybe a good place to add the 'retry' button would be the status-bubble's tool-tip (visible only if the bubble is red) https://webkit.org/b/196599 <https://webkit.org/b/196599> 4) Parse the relevant build failure message from build logs (and display in summary page) https://webkit.org/b/201941 <https://webkit.org/b/201941> 5) Style failure should be displayed in-line on the review page along-with the code, just like the reviewer's comments https://webkit.org/b/202252 <https://webkit.org/b/202252> 6) Add more test-suites to EWS (e.g.: LLDB tests, resultsdbpy tests) https://webkit.org/b/189206 <https://webkit.org/b/189206>, https://webkit.org/b/201928 <https://webkit.org/b/201928> 7) Add commit-queue support for security bugs https://webkit.org/b/201939 <https://webkit.org/b/201939> 8) API tests should upload crashlogs https://webkit.org/b/201929 <https://webkit.org/b/201929> Thanks Aakash
Hi Aakash, Thanks for doing this work.
On Sep 26, 2019, at 11:27 AM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I have been making number of improvements to EWS. I also have various planned improvements to EWS. I wanted to reach out to you guys to see if anyone wants me to prioritize any particular improvement(s). If there is any improvement which you want to see and is not listed below, please feel free to let me know. Also most of the queues have been transitioned from old to new EWS and I am working on the remaining ones (jsc, windows and commit-queue).
Here is the list of improvements (in no particular order):
1) Develop a webpage showing summary of EWS builds for a patch. This page would provide the summary of important build-steps, high-level details about the failure (e.g.: name of the tests which failed, or possibly relevant build failure logs), and include link(s) to the Buildbot page(s). This page will open on clicking the status-bubbles (and would be replacement of old EWS status page like https://webkit-queues.webkit.org/patch/379563/win-ews <https://webkit-queues.webkit.org/patch/379563/win-ews>). Currently clicking the status-bubble opens the Buildbot build page, which contains a lot of infrastructure details, and probably is information-overload for many engineers, so this summary page should help with that. https://webkit.org/b/197522 <https://webkit.org/b/197522>
2) Redesign status-bubble tooltip to include more detailed information about failures (e.g.: each test failure name along-with url to flakiness dashboard, and url to complete results.html file, as suggested by David Kilzer in https://lists.webkit.org/pipermail/webkit-dev/2019-September/030799.html <https://lists.webkit.org/pipermail/webkit-dev/2019-September/030799.html>). We should also add the tooltip support for iPad/iPhone https://webkit.org/b/201940 <https://webkit.org/b/201940>
3) Add a way to retry a patch in EWS. This would allow engineers to confirm that the failures indicated by EWS aren't flaky/incorrect. Maybe a good place to add the 'retry' button would be the status-bubble's tool-tip (visible only if the bubble is red) https://webkit.org/b/196599 <https://webkit.org/b/196599>This would be amazing. (My 2c: I'd vote for the button being visible without tooltip.)
- Saam
4) Parse the relevant build failure message from build logs (and display in summary page) https://webkit.org/b/201941 <https://webkit.org/b/201941>
5) Style failure should be displayed in-line on the review page along-with the code, just like the reviewer's comments https://webkit.org/b/202252 <https://webkit.org/b/202252>
6) Add more test-suites to EWS (e.g.: LLDB tests, resultsdbpy tests) https://webkit.org/b/189206 <https://webkit.org/b/189206>, https://webkit.org/b/201928 <https://webkit.org/b/201928>
7) Add commit-queue support for security bugs https://webkit.org/b/201939 <https://webkit.org/b/201939>
8) API tests should upload crashlogs https://webkit.org/b/201929 <https://webkit.org/b/201929>
Thanks Aakash _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
On Fri, Sep 27, 2019 at 7:57 AM Saam Barati <sbarati@apple.com> wrote:
Hi Aakash,
Thanks for doing this work.
On Sep 26, 2019, at 11:27 AM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I have been making number of improvements to EWS. I also have various planned improvements to EWS. I wanted to reach out to you guys to see if anyone wants me to prioritize any particular improvement(s). If there is any improvement which you want to see and is not listed below, please feel free to let me know. Also most of the queues have been transitioned from old to new EWS and I am working on the remaining ones (jsc, windows and commit-queue).
Here is the list of improvements (in no particular order):
1) Develop a webpage showing summary of EWS builds for a patch. This page would provide the summary of important build-steps, high-level details about the failure (e.g.: name of the tests which failed, or possibly relevant build failure logs), and include link(s) to the Buildbot page(s). This page will open on clicking the status-bubbles (and would be replacement of old EWS status page like https://webkit-queues.webkit.org/patch/379563/win-ews). Currently clicking the status-bubble opens the Buildbot build page, which contains a lot of infrastructure details, and probably is information-overload for many engineers, so this summary page should help with that. https://webkit.org/b/197522
2) Redesign status-bubble tooltip to include more detailed information about failures (e.g.: each test failure name along-with url to flakiness dashboard, and url to complete results.html file, as suggested by David Kilzer in https://lists.webkit.org/pipermail/webkit-dev/2019-September/030799.html). We should also add the tooltip support for iPad/iPhone https://webkit.org/b/201940
3) Add a way to retry a patch in EWS. This would allow engineers to confirm that the failures indicated by EWS aren't flaky/incorrect. Maybe a good place to add the 'retry' button would be the status-bubble's tool-tip (visible only if the bubble is red) https://webkit.org/b/196599
This would be amazing. (My 2c: I'd vote for the button being visible without tooltip.)
Yeah, this would be the most useful addition to EWS. 4) Parse the relevant build failure message from build logs (and display in
summary page) https://webkit.org/b/201941
5) Style failure should be displayed in-line on the review page along-with the code, just like the reviewer's comments https://webkit.org/b/202252
6) Add more test-suites to EWS (e.g.: LLDB tests, resultsdbpy tests) https://webkit.org/b/189206, https://webkit.org/b/201928
7) Add commit-queue support for security bugs https://webkit.org/b/201939
8) API tests should upload crashlogs https://webkit.org/b/201929
Along with this one. - R. Niwa -- - R. Niwa
Hi Everyone, I am very happy to announce that I have added the ability to retry the patches in EWS :) From now on, for any patch, whenever any of the EWS status-bubble is red, you will see another button named 'Retry failed builds' next to the status-bubbles. Clicking on 'Retry failed builds' button would cause all the failed EWS builds for that patch to be retried. Please click on this button only when needed, as unnecessary retries might put load on the system and others patches might have to wait longer. As always, please feel free to let me know if you notice any issue or have any feedback. Known UI issues: https://bugs.webkit.org/show_bug.cgi?id=203249 <https://bugs.webkit.org/show_bug.cgi?id=203249> Thanks Aakash
On Sep 26, 2019, at 7:56 PM, Saam Barati <sbarati@apple.com> wrote:
Hi Aakash,
Thanks for doing this work.
On Sep 26, 2019, at 11:27 AM, Aakash Jain <aakash_jain@apple.com <mailto:aakash_jain@apple.com>> wrote:
...
3) Add a way to retry a patch in EWS. This would allow engineers to confirm that the failures indicated by EWS aren't flaky/incorrect. Maybe a good place to add the 'retry' button would be the status-bubble's tool-tip (visible only if the bubble is red) https://webkit.org/b/196599 <https://webkit.org/b/196599>This would be amazing. (My 2c: I'd vote for the button being visible without tooltip.)
- Saam
On Sep 27, 2019, at 3:27 AM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I have been making number of improvements to EWS. I also have various planned improvements to EWS. I wanted to reach out to you guys to see if anyone
Not everyone on this list is a guy.
wants me to prioritize any particular improvement(s). If there is any improvement which you want to see and is not listed below, please feel free to let me know. Also most of the queues have been transitioned from old to new EWS and I am working on the remaining ones (jsc, windows and commit-queue).
Here is the list of improvements (in no particular order):
1) Develop a webpage showing summary of EWS builds for a patch. This page would provide the summary of important build-steps, high-level details about the failure (e.g.: name of the tests which failed, or possibly relevant build failure logs), and include link(s) to the Buildbot page(s). This page will open on clicking the status-bubbles (and would be replacement of old EWS status page like https://webkit-queues.webkit.org/patch/379563/win-ews <https://webkit-queues.webkit.org/patch/379563/win-ews>). Currently clicking the status-bubble opens the Buildbot build page, which contains a lot of infrastructure details, and probably is information-overload for many engineers, so this summary page should help with that. https://webkit.org/b/197522 <https://webkit.org/b/197522>
2) Redesign status-bubble tooltip to include more detailed information about failures (e.g.: each test failure name along-with url to flakiness dashboard, and url to complete results.html file, as suggested by David Kilzer in https://lists.webkit.org/pipermail/webkit-dev/2019-September/030799.html <https://lists.webkit.org/pipermail/webkit-dev/2019-September/030799.html>). We should also add the tooltip support for iPad/iPhone https://webkit.org/b/201940 <https://webkit.org/b/201940>
#1
3) Add a way to retry a patch in EWS. This would allow engineers to confirm that the failures indicated by EWS aren't flaky/incorrect. Maybe a good place to add the 'retry' button would be the status-bubble's tool-tip (visible only if the bubble is red) https://webkit.org/b/196599 <https://webkit.org/b/196599>
#2
4) Parse the relevant build failure message from build logs (and display in summary page) https://webkit.org/b/201941 <https://webkit.org/b/201941>
5) Style failure should be displayed in-line on the review page along-with the code, just like the reviewer's comments https://webkit.org/b/202252 <https://webkit.org/b/202252>
6) Add more test-suites to EWS (e.g.: LLDB tests, resultsdbpy tests) https://webkit.org/b/189206 <https://webkit.org/b/189206>, https://webkit.org/b/201928 <https://webkit.org/b/201928>
7) Add commit-queue support for security bugs https://webkit.org/b/201939 <https://webkit.org/b/201939>
8) API tests should upload crashlogs https://webkit.org/b/201929 <https://webkit.org/b/201929>
Simon
27.09.2019, 08:15, "Simon Fraser" <simon.fraser@apple.com>:
On Sep 27, 2019, at 3:27 AM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I have been making number of improvements to EWS. I also have various planned improvements to EWS. I wanted to reach out to you guys to see if anyone
Not everyone on this list is a guy.
FWIW, plural form of "guy" is often used as gender-neutral vocative, see e.g. [1,2] [1] https://www.merriam-webster.com/dictionary/guy [2] https://dictionary.cambridge.org/us/dictionary/english/guy -- Regards, Konstantin
Thank You everyone (who responded) for the feedback. I will prioritize the features some of you mentioned, especially adding a way to retry the patch. Thanks Aakash
On Sep 27, 2019, at 9:27 AM, Konstantin Tokarev <annulen@yandex.ru> wrote:
27.09.2019, 08:15, "Simon Fraser" <simon.fraser@apple.com>:
On Sep 27, 2019, at 3:27 AM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I have been making number of improvements to EWS. I also have various planned improvements to EWS. I wanted to reach out to you guys to see if anyone
Not everyone on this list is a guy.
FWIW, plural form of "guy" is often used as gender-neutral vocative, see e.g. [1,2]
[1] https://www.merriam-webster.com/dictionary/guy [2] https://dictionary.cambridge.org/us/dictionary/english/guy
-- Regards, Konstantin
On Sep 27, 2019, at 10:27 PM, Konstantin Tokarev <annulen@yandex.ru> wrote:
27.09.2019, 08:15, "Simon Fraser" <simon.fraser@apple.com>:
On Sep 27, 2019, at 3:27 AM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I have been making number of improvements to EWS. I also have various planned improvements to EWS. I wanted to reach out to you guys to see if anyone
Not everyone on this list is a guy.
FWIW, plural form of "guy" is often used as gender-neutral vocative, see e.g. [1,2]
[1] https://www.merriam-webster.com/dictionary/guy [2] https://dictionary.cambridge.org/us/dictionary/english/guy
I apologize to Aakash for calling this out publicly. However, ‘guys’ clearly has gender-specific roots, and to my ears, it sounds male-biased, There are many other words that can be used in its place which are more inclusive (“folks”, “everyone”, “you all” etc). Simon
Hello Aakash (and everybody else), On Thu, 26 Sep 2019 14:27:25 -0400, Aakash Jain <aakash_jain@apple.com> wrote:
I have been making number of improvements to EWS. I also have various planned improvements to EWS […]
Thanks once again for your continued work on improving the EWS, it is much appreciated =)
Here is the list of improvements (in no particular order):
[…]
3) Add a way to retry a patch in EWS. This would allow engineers to confirm that the failures indicated by EWS aren't flaky/incorrect. Maybe a good place to add the 'retry' button would be the status-bubble's tool-tip (visible only if the bubble is red) https://webkit.org/b/196599 <https://webkit.org/b/196599>
Being able to retry a patch without having to re-upload it, and more importantly without having to ping another person, would be neat. I don't have a strong opinion about which tasks to prioritize first, tho.
[…]
6) Add more test-suites to EWS (e.g.: LLDB tests, resultsdbpy tests) https://webkit.org/b/189206 <https://webkit.org/b/189206>, https://webkit.org/b/201928 <https://webkit.org/b/201928>
In this line of adding more things that the EWS could test, among the developers of the GTK/WPE ports we think it would be manageable for us to have bots run API tests. There is some work we need to do, and the intention is to start with GTK first, see how it goes, and then add WPE afterwards. Carlos opened a bug for this, I took the liberty of CC'ing you assuming that you might want to at least keep tabs on it (feel free to unsubscribe): https://bugs.webkit.org/show_bug.cgi?id=202361 Best regards, —Adrián
On Sep 30, 2019, at 9:32 AM, Adrian Perez de Castro <aperez@igalia.com> wrote:
Hello Aakash (and everybody else),
On Thu, 26 Sep 2019 14:27:25 -0400, Aakash Jain <aakash_jain@apple.com> wrote:
I have been making number of improvements to EWS. I also have various planned improvements to EWS […]
Thanks once again for your continued work on improving the EWS, it is much appreciated =)
Thank You for the feedback. I am hoping that Improved EWS would increase overall team's productivity as people have to deal less with build/test breakages and rollouts.
Here is the list of improvements (in no particular order):
[…]
3) Add a way to retry a patch in EWS. This would allow engineers to confirm that the failures indicated by EWS aren't flaky/incorrect. Maybe a good place to add the 'retry' button would be the status-bubble's tool-tip (visible only if the bubble is red) https://webkit.org/b/196599 <https://webkit.org/b/196599>
Being able to retry a patch without having to re-upload it, and more importantly without having to ping another person, would be neat. I don't have a strong opinion about which tasks to prioritize first, tho.
[…]
6) Add more test-suites to EWS (e.g.: LLDB tests, resultsdbpy tests) https://webkit.org/b/189206 <https://webkit.org/b/189206>, https://webkit.org/b/201928 <https://webkit.org/b/201928>
In this line of adding more things that the EWS could test, among the developers of the GTK/WPE ports we think it would be manageable for us to have bots run API tests. There is some work we need to do, and the intention is to start with GTK first, see how it goes, and then add WPE afterwards. Carlos opened a bug for this, I took the liberty of CC'ing you assuming that you might want to at least keep tabs on it (feel free to unsubscribe):
Sure, I will work with you and Carlos on this.
Best regards, —Adrián
participants (6)
-
Aakash Jain
-
Adrian Perez de Castro
-
Konstantin Tokarev
-
Ryosuke Niwa
-
Saam Barati
-
Simon Fraser