Introducing brand new EWS with EWS for API Tests (for macOS and iOS) and EWS for WebKitPerl Tests! Starting today, when you upload a patch to bugs.webkit.org <http://bugs.webkit.org/>, you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag). The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS. Why new EWS? The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware. The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware. The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org <http://build.webkit.org/>). How can you contribute: If you are interested in contributing, the source code is located at: ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build ews-app (web-app): Tools/BuildSlaveSupport/ews-app Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSy... <https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSystem> Upcoming features: - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 <https://bugs.webkit.org/show_bug.cgi?id=196607> - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 <https://bugs.webkit.org/show_bug.cgi?id=196598> - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 <https://bugs.webkit.org/show_bug.cgi?id=196599> - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605 <https://bugs.webkit.org/show_bug.cgi?id=196605> If you notice any issue, please feel free to file bugs (and assign to me). Thanks & Regards Aakash Jain
The new bubbles seem to be working well, and having API tests running in EWS is great! Can we add the waterfall view to https://ews-build.webkit.org/#/builders so that we can see where our patches are in the queue? - R. Niwa On Thu, Apr 4, 2019 at 6:59 PM Aakash Jain <aakash_jain@apple.com> wrote:
Introducing brand new EWS
with EWS for API Tests (for macOS and iOS)
and EWS for WebKitPerl Tests!
Starting today, when you upload a patch to bugs.webkit.org, you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag).
The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS.
*Why new EWS?* The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware.
The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware.
The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org).
*How can you contribute:* If you are interested in contributing, the source code is located at: ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build ews-app (web-app): Tools/BuildSlaveSupport/ews-app
Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSy...
*Upcoming features:* - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605
If you notice any issue, please feel free to file bugs (and assign to me).
Thanks & Regards Aakash Jain _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Hi Ryosuke, Sorry for the delay. I will soon be adding the feature to display patch's position in queue in https://bugs.webkit.org/show_bug.cgi?id=196607 <https://bugs.webkit.org/show_bug.cgi?id=196607>. Would that address your concern? Thanks Aakash
On Apr 4, 2019, at 10:09 PM, Ryosuke Niwa <rniwa@webkit.org> wrote:
The new bubbles seem to be working well, and having API tests running in EWS is great!
Can we add the waterfall view to https://ews-build.webkit.org/#/builders <https://ews-build.webkit.org/#/builders> so that we can see where our patches are in the queue?
- R. Niwa
On Thu, Apr 4, 2019 at 6:59 PM Aakash Jain <aakash_jain@apple.com <mailto:aakash_jain@apple.com>> wrote: Introducing brand new EWS
with EWS for API Tests (for macOS and iOS)
and EWS for WebKitPerl Tests!
Starting today, when you upload a patch to bugs.webkit.org <http://bugs.webkit.org/>, you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag).
The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS.
Why new EWS? The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware.
The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware.
The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org <http://build.webkit.org/>).
How can you contribute: If you are interested in contributing, the source code is located at: ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build ews-app (web-app): Tools/BuildSlaveSupport/ews-app
Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSy... <https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSystem>
Upcoming features: - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 <https://bugs.webkit.org/show_bug.cgi?id=196607> - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 <https://bugs.webkit.org/show_bug.cgi?id=196598> - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 <https://bugs.webkit.org/show_bug.cgi?id=196599> - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605 <https://bugs.webkit.org/show_bug.cgi?id=196605>
If you notice any issue, please feel free to file bugs (and assign to me).
Thanks & Regards Aakash Jain _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
Yes, that would be great. Although I'd still prefer having a proper waterfall view so that when my patch isn't getting processed / very behind in the queue, I can figure out what's causing it on my own instead of having to tell a bot watcher something is wrong with EWS. - R. Niwa On Thu, Apr 11, 2019 at 12:17 PM Aakash Jain <aakash_jain@apple.com> wrote:
Hi Ryosuke,
Sorry for the delay. I will soon be adding the feature to display patch's position in queue in https://bugs.webkit.org/show_bug.cgi?id=196607. Would that address your concern?
Thanks Aakash
On Apr 4, 2019, at 10:09 PM, Ryosuke Niwa <rniwa@webkit.org> wrote:
The new bubbles seem to be working well, and having API tests running in EWS is great!
Can we add the waterfall view to https://ews-build.webkit.org/#/builders so that we can see where our patches are in the queue?
- R. Niwa
On Thu, Apr 4, 2019 at 6:59 PM Aakash Jain <aakash_jain@apple.com> wrote:
Introducing brand new EWS
with EWS for API Tests (for macOS and iOS)
and EWS for WebKitPerl Tests!
Starting today, when you upload a patch to bugs.webkit.org, you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag).
The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS.
*Why new EWS?* The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware.
The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware.
The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org).
*How can you contribute:* If you are interested in contributing, the source code is located at: ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build ews-app (web-app): Tools/BuildSlaveSupport/ews-app
Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSy...
*Upcoming features:* - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605
If you notice any issue, please feel free to file bugs (and assign to me).
Thanks & Regards Aakash Jain _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Hi Everyone, I just wanted to update everyone with the recent improvements I have made to new EWS. As always, please feel encouraged to provide any feedback (either by filing bugs or contacting me directly). New Features: EWS status-bubble now display position in queue while patch is waiting to be processed Added webkitpy and bindings-tests EWS (moved from old to new EWS) Status bubbles for webkitpy and bindings-tests EWS now display the exact test failures in hover-over message (https://webkit.org/b/197395 <https://webkit.org/b/197395>, https://webkit.org/b/197423 <https://webkit.org/b/197423>) Added support for 'new EWS' in webkit-patch tool Added 'EWS Build Archives' (similar to 'WebKit Build Archives' https://webkit.org/blog/7978/introducing-webkit-build-archives/ <https://webkit.org/blog/7978/introducing-webkit-build-archives/>). For every patch uploaded to Bugzilla, EWS builders build the patch for various platforms (currently macOS and iOS) and upload the archives to S3. These archives are available to download by anyone (for 14 days). The S3 URL is in corresponding build (e.g.: notice 'uploaded archive' link in https://ews-build.webkit.org/#/builders/7/builds/2477 <https://ews-build.webkit.org/#/builders/7/builds/2477>). So, if for any reason, you want to get a built archive for your patch, you can simply upload the patch to Bugzilla. (Note that if there is interest in this, we can enhance it further) Infrastructure Improvements: Flakiness in API tests has been reduced (thanks to many WebKit developers) Infrastructure improvements to prevent build failure due to "worker not pinged" (e.g.: https://ews-build.webkit.org/#/builders/9/builds/332 <https://ews-build.webkit.org/#/builders/9/builds/332>) New EWS polls bugzilla more frequently https://webkit.org/b/197138 <https://webkit.org/b/197138> Configured DEBUG mode appropriately for Production and Development env https://webkit.org/b/197700 <https://webkit.org/b/197700> Ensured that Buildbot worker logs are not lost on restarting worker Do not run clean build by default on EWS builders (to improve efficiency) https://webkit.org/b/196897 <https://webkit.org/b/196897> build.webkit.org <http://build.webkit.org/> and ews-build.webkit.org <http://ews-build.webkit.org/> starting sharing code (although very little as of now, however the plan is to share more code) Added migrations file to repository https://webkit.org/b/197729 <https://webkit.org/b/197729> Added EWS bots information to Internal scripts to easily monitor bots Added more unit-tests Bug fixes: Clicking 'submit to new ews' doesn't reload status-bubble https://webkit.org/b/196675 <https://webkit.org/b/196675> Clicking on white bubble navigates to page with only bubbles https://webkit.org/b/197520 <https://webkit.org/b/197520> Submit to EWS buttons are not aligned properly with status-bubbles https://webkit.org/b/197139 <https://webkit.org/b/197139> Status bubble should turn orange when any build step fails https://webkit.org/b/197812 <https://webkit.org/b/197812> Handle bug titles with unicode characters https://webkit.org/b/196802 <https://webkit.org/b/196802> Scripts using Buildbot API have CORS error https://webkit.org/b/196709 <https://webkit.org/b/196709> PrintConfiguration should display Xcode version instead of SDKVersion https://webkit.org/b/196780 <https://webkit.org/b/196780> Trigger queues only after uploading the archive https://webkit.org/b/197180 <https://webkit.org/b/197180> Do not upload archive when Compile Fails https://webkit.org/b/196674 <https://webkit.org/b/196674> Exception while loading status-bubble when no build step has started https://webkit.org/b/196676 <https://webkit.org/b/196676> Use singular verb in failure description in case of single api test failure https://webkit.org/b/197013 <https://webkit.org/b/197013> EWS should clearly indicate flaky test failures https://webkit.org/b/196947 <https://webkit.org/b/196947> Use explicit imports instead of wildcard imports https://webkit.org/b/197194 <https://webkit.org/b/197194> New EWS: patches on recently added queues listed as #1 for older bugs https://webkit.org/b/197496 <https://webkit.org/b/197496> Improved summary text for various build steps Interesting info: Since last month, 'EWS for API tests' prevented API test breakage on 50+ patches (https://ews-build.webkit.org/api/v2/builders/3/builds?state_string__contains... <https://ews-build.webkit.org/api/v2/builders/3/builds?state_string__contains=new%20API%20Test&property=bug_id&order=-number>). Thanks Aakash
On Apr 4, 2019, at 10:00 PM, Aakash Jain <aakash_jain@apple.com> wrote:
Introducing brand new EWS
with EWS for API Tests (for macOS and iOS)
and EWS for WebKitPerl Tests!
Starting today, when you upload a patch to bugs.webkit.org <http://bugs.webkit.org/>, you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag).
The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS.
Why new EWS? The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware.
The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware.
The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org <http://build.webkit.org/>).
How can you contribute: If you are interested in contributing, the source code is located at: ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build ews-app (web-app): Tools/BuildSlaveSupport/ews-app
Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSy... <https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSystem>
Upcoming features: - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 <https://bugs.webkit.org/show_bug.cgi?id=196607> - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 <https://bugs.webkit.org/show_bug.cgi?id=196598> - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 <https://bugs.webkit.org/show_bug.cgi?id=196599> - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605 <https://bugs.webkit.org/show_bug.cgi?id=196605>
If you notice any issue, please feel free to file bugs (and assign to me).
Thanks & Regards Aakash Jain
Hi Everyone, I want to update everyone with the further improvements I have made to new EWS. As always, please feel encouraged to provide any feedback (either by filing bugs or contacting me directly). New Features: - Launched 'Security EWS' - Added iOS-12 Builder queue on new EWS (moved from old to new EWS) - Added WPE and GTK queues on new EWS (moved from old to new EWS) - New EWS can now process large patches (larger than 640kb) https://webkit.org/b/198851 <https://webkit.org/b/198851> Infrastructure Improvements: - EWS now automatically retries builds in case of certain infrastructure issues and ToT compile failures https://bugs.webkit.org/show_bug.cgi?id=199025 <https://bugs.webkit.org/show_bug.cgi?id=199025> - Improved stability by adding KillOldProcesses step before API and Layout tests https://webkit.org/b/199592 <https://webkit.org/b/199592> - Shared bots across queues to improve efficiency https://webkit.org/b/198370 <https://webkit.org/b/198370> - Added check for duplicate workers in config.json https://webkit.org/b/199240 <https://webkit.org/b/199240> - Patch link now opens the pretty-patch url https://webkit.org/b/199031 <https://webkit.org/b/199031> - Triggered builds should use same revision as parent build (fixes a corner case) https://webkit.org/b/198289 <https://webkit.org/b/198289> - Display pre-existing API test failures in build summary to help bot-watchers https://webkit.org/b/199525 <https://webkit.org/b/199525> - Send email notifications for failures (for maintenance/bot-watching) https://webkit.org/b/198919 <https://webkit.org/b/198919> - Remove unused buildbot tabs for better readability https://webkit.org/b/198108 <https://webkit.org/b/198108> - Allow skipping uploading built product for few builders https://webkit.org/b/199422 <https://webkit.org/b/199422> - EWS should provide option to download layout test results zip file https://webkit.org/b/199121 <https://webkit.org/b/199121> - Retry Layout test in case of failures https://webkit.org/b/199194 <https://webkit.org/b/199194> - Upload test results after running layout-tests https://webkit.org/b/199120 <https://webkit.org/b/199120> - Improved error message on performing certain actions (like Rebuild) which requires authentication - Added more unit-tests Bug fixes: - Results are clobbered in UploadTestResults and ExtractTestResults steps in case of multiple layout test runs https://webkit.org/b/199178 <https://webkit.org/b/199178> - New EWS: api-ios does not differentiate between patch compile failure and ToT compile failure https://webkit.org/b/197850 <https://webkit.org/b/197850> - Buildbot worker not pinged https://webkit.org/b/199438 <https://webkit.org/b/199438> - Make PrintConfiguration platform aware https://webkit.org/b/196657 <https://webkit.org/b/196657> - Status bubble should not turn orange when any build step is skipped https://webkit.org/b/199079 <https://webkit.org/b/199079> - Do not print worker environment variables in each build step https://webkit.org/b/197319 <https://webkit.org/b/197319> - Do not run unix commands for windows in PrintConfiguration https://webkit.org/b/199605 <https://webkit.org/b/199605> Thanks Aakash
On May 22, 2019, at 7:36 PM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I just wanted to update everyone with the recent improvements I have made to new EWS. As always, please feel encouraged to provide any feedback (either by filing bugs or contacting me directly).
New Features: EWS status-bubble now display position in queue while patch is waiting to be processed Added webkitpy and bindings-tests EWS (moved from old to new EWS) Status bubbles for webkitpy and bindings-tests EWS now display the exact test failures in hover-over message (https://webkit.org/b/197395 <https://webkit.org/b/197395>, https://webkit.org/b/197423 <https://webkit.org/b/197423>) Added support for 'new EWS' in webkit-patch tool Added 'EWS Build Archives' (similar to 'WebKit Build Archives' https://webkit.org/blog/7978/introducing-webkit-build-archives/ <https://webkit.org/blog/7978/introducing-webkit-build-archives/>). For every patch uploaded to Bugzilla, EWS builders build the patch for various platforms (currently macOS and iOS) and upload the archives to S3. These archives are available to download by anyone (for 14 days). The S3 URL is in corresponding build (e.g.: notice 'uploaded archive' link in https://ews-build.webkit.org/#/builders/7/builds/2477 <https://ews-build.webkit.org/#/builders/7/builds/2477>). So, if for any reason, you want to get a built archive for your patch, you can simply upload the patch to Bugzilla. (Note that if there is interest in this, we can enhance it further)
Infrastructure Improvements: Flakiness in API tests has been reduced (thanks to many WebKit developers) Infrastructure improvements to prevent build failure due to "worker not pinged" (e.g.: https://ews-build.webkit.org/#/builders/9/builds/332 <https://ews-build.webkit.org/#/builders/9/builds/332>) New EWS polls bugzilla more frequently https://webkit.org/b/197138 <https://webkit.org/b/197138> Configured DEBUG mode appropriately for Production and Development env https://webkit.org/b/197700 <https://webkit.org/b/197700> Ensured that Buildbot worker logs are not lost on restarting worker Do not run clean build by default on EWS builders (to improve efficiency) https://webkit.org/b/196897 <https://webkit.org/b/196897> build.webkit.org <http://build.webkit.org/> and ews-build.webkit.org <http://ews-build.webkit.org/> starting sharing code (although very little as of now, however the plan is to share more code) Added migrations file to repository https://webkit.org/b/197729 <https://webkit.org/b/197729> Added EWS bots information to Internal scripts to easily monitor bots Added more unit-tests
Bug fixes: Clicking 'submit to new ews' doesn't reload status-bubble https://webkit.org/b/196675 <https://webkit.org/b/196675> Clicking on white bubble navigates to page with only bubbles https://webkit.org/b/197520 <https://webkit.org/b/197520> Submit to EWS buttons are not aligned properly with status-bubbles https://webkit.org/b/197139 <https://webkit.org/b/197139> Status bubble should turn orange when any build step fails https://webkit.org/b/197812 <https://webkit.org/b/197812> Handle bug titles with unicode characters https://webkit.org/b/196802 <https://webkit.org/b/196802> Scripts using Buildbot API have CORS error https://webkit.org/b/196709 <https://webkit.org/b/196709> PrintConfiguration should display Xcode version instead of SDKVersion https://webkit.org/b/196780 <https://webkit.org/b/196780> Trigger queues only after uploading the archive https://webkit.org/b/197180 <https://webkit.org/b/197180> Do not upload archive when Compile Fails https://webkit.org/b/196674 <https://webkit.org/b/196674> Exception while loading status-bubble when no build step has started https://webkit.org/b/196676 <https://webkit.org/b/196676> Use singular verb in failure description in case of single api test failure https://webkit.org/b/197013 <https://webkit.org/b/197013> EWS should clearly indicate flaky test failures https://webkit.org/b/196947 <https://webkit.org/b/196947> Use explicit imports instead of wildcard imports https://webkit.org/b/197194 <https://webkit.org/b/197194> New EWS: patches on recently added queues listed as #1 for older bugs https://webkit.org/b/197496 <https://webkit.org/b/197496> Improved summary text for various build steps
Interesting info: Since last month, 'EWS for API tests' prevented API test breakage on 50+ patches (https://ews-build.webkit.org/api/v2/builders/3/builds?state_string__contains... <https://ews-build.webkit.org/api/v2/builders/3/builds?state_string__contains=new%20API%20Test&property=bug_id&order=-number>).
Thanks Aakash
On Apr 4, 2019, at 10:00 PM, Aakash Jain <aakash_jain@apple.com <mailto:aakash_jain@apple.com>> wrote:
Introducing brand new EWS
with EWS for API Tests (for macOS and iOS)
and EWS for WebKitPerl Tests!
Starting today, when you upload a patch to bugs.webkit.org <http://bugs.webkit.org/>, you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag).
The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS.
Why new EWS? The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware.
The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware.
The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org <http://build.webkit.org/>).
How can you contribute: If you are interested in contributing, the source code is located at: ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build ews-app (web-app): Tools/BuildSlaveSupport/ews-app
Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSy... <https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSystem>
Upcoming features: - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 <https://bugs.webkit.org/show_bug.cgi?id=196607> - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 <https://bugs.webkit.org/show_bug.cgi?id=196598> - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 <https://bugs.webkit.org/show_bug.cgi?id=196599> - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605 <https://bugs.webkit.org/show_bug.cgi?id=196605>
If you notice any issue, please feel free to file bugs (and assign to me).
Thanks & Regards Aakash Jain
Hi Everyone, I want to update everyone with the further improvements I have made to new EWS. As always, please feel free to provide any feedback (either by filing bugs or contacting me directly). New Features: - Moved following queues to from old to new EWS: • iOS 12 Tests EWS • macOS WK1 and WK2 Release Tests EWS • macOS WK1 Debug Tests EWS • GTK and WPE EWS • WinCairo EWS • Style (and watchlist) EWS - Added a new 'services' EWS to run Buildbot unit tests https://webkit.org/b/199994 - Separate status bubbles for builders and layout-testers Infrastructure Improvements: - EWS is now able to automatically recover (and retry the build) in case of certain git and network errors https://webkit.org/b/199722 - Some EWS bots now reboot periodically to ensure that start from clean state - Added KillOldProcesses step before running API or Layout tests to increase robustness https://webkit.org/b/199592 - 'view layout test results' option is now displayed next to layout-test build step https://webkit.org/b/200048 - EWS now displays pre-existing Layout test failure names in the build summary https://webkit.org/b/199941 - Machine uptime is now reported in PrintConfiguration step https://webkit.org/b/200812 - Added EWS Buildbot support to Internal scripts to easily find/reconfigure bots - Flakiness in API tests has been further reduced (thanks to multiple fixes by WebKit developers) - Added more unit tests Bug fixes: - EWS status-bubbles are sometimes multi-row with scroll-bar https://webkit.org/b/199939 - EWS: Cannot see build status page when patch is waiting for tester https://webkit.org/b/200333 - EWS: Do not append additional '(failure)' string at the end of custom failure message in EWS Buildbot https://webkit.org/b/201140 - [ews] Status bubble should display only important messages in pop-over https://webkit.org/b/201308 Only remaining queues on old EWS are windows, jsc and commit-queue, which I will be working on next. Thanks Aakash
On Jul 11, 2019, at 10:18 am, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I want to update everyone with the further improvements I have made to new EWS. As always, please feel encouraged to provide any feedback (either by filing bugs or contacting me directly).
New Features: - Launched 'Security EWS' - Added iOS-12 Builder queue on new EWS (moved from old to new EWS) - Added WPE and GTK queues on new EWS (moved from old to new EWS) - New EWS can now process large patches (larger than 640kb) https://webkit.org/b/198851
Infrastructure Improvements: - EWS now automatically retries builds in case of certain infrastructure issues and ToT compile failures https://bugs.webkit.org/show_bug.cgi?id=199025 - Improved stability by adding KillOldProcesses step before API and Layout tests https://webkit.org/b/199592 - Shared bots across queues to improve efficiency https://webkit.org/b/198370 - Added check for duplicate workers in config.json https://webkit.org/b/199240 - Patch link now opens the pretty-patch url https://webkit.org/b/199031 - Triggered builds should use same revision as parent build (fixes a corner case) https://webkit.org/b/198289 - Display pre-existing API test failures in build summary to help bot-watchers https://webkit.org/b/199525 - Send email notifications for failures (for maintenance/bot-watching) https://webkit.org/b/198919 - Remove unused buildbot tabs for better readability https://webkit.org/b/198108 - Allow skipping uploading built product for few builders https://webkit.org/b/199422 - EWS should provide option to download layout test results zip file https://webkit.org/b/199121 - Retry Layout test in case of failures https://webkit.org/b/199194 - Upload test results after running layout-tests https://webkit.org/b/199120 - Improved error message on performing certain actions (like Rebuild) which requires authentication - Added more unit-tests
Bug fixes: - Results are clobbered in UploadTestResults and ExtractTestResults steps in case of multiple layout test runs https://webkit.org/b/199178 - New EWS: api-ios does not differentiate between patch compile failure and ToT compile failure https://webkit.org/b/197850 - Buildbot worker not pinged https://webkit.org/b/199438 - Make PrintConfiguration platform aware https://webkit.org/b/196657 - Status bubble should not turn orange when any build step is skipped https://webkit.org/b/199079 - Do not print worker environment variables in each build step https://webkit.org/b/197319 - Do not run unix commands for windows in PrintConfiguration https://webkit.org/b/199605
Thanks Aakash
On May 22, 2019, at 7:36 PM, Aakash Jain <aakash_jain@apple.com> wrote:
Hi Everyone,
I just wanted to update everyone with the recent improvements I have made to new EWS. As always, please feel encouraged to provide any feedback (either by filing bugs or contacting me directly).
New Features: • EWS status-bubble now display position in queue while patch is waiting to be processed • Added webkitpy and bindings-tests EWS (moved from old to new EWS) • Status bubbles for webkitpy and bindings-tests EWS now display the exact test failures in hover-over message (https://webkit.org/b/197395, https://webkit.org/b/197423) • Added support for 'new EWS' in webkit-patch tool • Added 'EWS Build Archives' (similar to 'WebKit Build Archives' https://webkit.org/blog/7978/introducing-webkit-build-archives/). For every patch uploaded to Bugzilla, EWS builders build the patch for various platforms (currently macOS and iOS) and upload the archives to S3. These archives are available to download by anyone (for 14 days). The S3 URL is in corresponding build (e.g.: notice 'uploaded archive' link in https://ews-build.webkit.org/#/builders/7/builds/2477). So, if for any reason, you want to get a built archive for your patch, you can simply upload the patch to Bugzilla. (Note that if there is interest in this, we can enhance it further)
Infrastructure Improvements: • Flakiness in API tests has been reduced (thanks to many WebKit developers) • Infrastructure improvements to prevent build failure due to "worker not pinged" (e.g.: https://ews-build.webkit.org/#/builders/9/builds/332) • New EWS polls bugzilla more frequently https://webkit.org/b/197138 • Configured DEBUG mode appropriately for Production and Development env https://webkit.org/b/197700 • Ensured that Buildbot worker logs are not lost on restarting worker • Do not run clean build by default on EWS builders (to improve efficiency) https://webkit.org/b/196897 • build.webkit.org and ews-build.webkit.org starting sharing code (although very little as of now, however the plan is to share more code) • Added migrations file to repository https://webkit.org/b/197729 • Added EWS bots information to Internal scripts to easily monitor bots • Added more unit-tests
Bug fixes: • Clicking 'submit to new ews' doesn't reload status-bubble https://webkit.org/b/196675 • Clicking on white bubble navigates to page with only bubbles https://webkit.org/b/197520 • Submit to EWS buttons are not aligned properly with status-bubbles https://webkit.org/b/197139 • Status bubble should turn orange when any build step fails https://webkit.org/b/197812 • Handle bug titles with unicode characters https://webkit.org/b/196802 • Scripts using Buildbot API have CORS error https://webkit.org/b/196709 • PrintConfiguration should display Xcode version instead of SDKVersion https://webkit.org/b/196780 • Trigger queues only after uploading the archive https://webkit.org/b/197180 • Do not upload archive when Compile Fails https://webkit.org/b/196674 • Exception while loading status-bubble when no build step has started https://webkit.org/b/196676 • Use singular verb in failure description in case of single api test failure https://webkit.org/b/197013 • EWS should clearly indicate flaky test failures https://webkit.org/b/196947 • Use explicit imports instead of wildcard imports https://webkit.org/b/197194 • New EWS: patches on recently added queues listed as #1 for older bugs https://webkit.org/b/197496 • Improved summary text for various build steps
Interesting info: Since last month, 'EWS for API tests' prevented API test breakage on 50+ patches (https://ews-build.webkit.org/api/v2/builders/3/builds?state_string__contains...).
Thanks Aakash
On Apr 4, 2019, at 10:00 PM, Aakash Jain <aakash_jain@apple.com> wrote:
Introducing brand new EWS
with EWS for API Tests (for macOS and iOS)
and EWS for WebKitPerl Tests!
Starting today, when you upload a patch to bugs.webkit.org, you will see few more bubbles (for API tests and webkitperl tests). You might also see additional button 'Submit to new EWS' (if the patch doesn't have r? flag).
The new EWS comes with many new features and there are lot more I want to add. But, I don't want you guys to wait more to start getting the benefits. That's why I am rolling it out in phases, starting with EWS for API Tests and WebKitPerl Tests. These are tests which are not currently covered by the existing EWS. Next step would be to move queues from existing EWS to new EWS one by one, with the eventual goal of moving over everything to new EWS.
Why new EWS? The existing EWS has certain architectural limitations. One of the prominent limitation is that there is no concept of building and testing the patch on different queues. If we have three queues: WK1 tests, WK2 tests and API tests, all three queues would need to compile WebKit independently. So WebKit would be compiled thrice instead of once. This is inefficient and thereby require more hardware.
The new EWS has separate builder and tester queues. Builder queues build once and upload the archive. Multiple tester queues download that same archive and run tests on that. That way WebKit is compiled only once, and re-used on multiple tester queues. This improves system efficiency and allows us to add new test queues with substantially less hardware.
The new EWS uses Buildbot at the back-end, which is a production-level CI system. It is easier to maintain and automatically provide various features like historical build logs, real-time log streaming, easier bot management, ability to retry a build etc. Plus, it’s a system most of you are already familiar with (build.webkit.org).
How can you contribute: If you are interested in contributing, the source code is located at: ews-build (Buildbot): Tools/BuildSlaveSupport/ews-build ews-app (web-app): Tools/BuildSlaveSupport/ews-app
Detailed instructions are at: https://trac.webkit.org/wiki/EarlyWarningSystem#ContributingtoEarlyWarningSy...
Upcoming features: - status-bubble should display position in queue https://bugs.webkit.org/show_bug.cgi?id=196607 - EWS should comment on Bugzilla bugs about failures https://bugs.webkit.org/show_bug.cgi?id=196598 - EWS should have a way to retry a patch https://bugs.webkit.org/show_bug.cgi?id=196599 - Security EWS https://bugs.webkit.org/show_bug.cgi?id=196605
If you notice any issue, please feel free to file bugs (and assign to me).
Thanks & Regards Aakash Jain
Hello Aakash, WebKittens, On Thu, 29 Aug 2019 18:01:56 -0400, Aakash Jain <aakash_jain@apple.com> wrote:
I want to update everyone with the further improvements I have made to new EWS. As always, please feel free to provide any feedback (either by filing bugs or contacting me directly).
New Features:
[...]
Thanks a ton for all the improvements in the EWS! Since we moved the GTK/EWS builders to the new queues they have been faster and more reliable. After solving a couple of minor hiccups right after the switch, the builders have not needed any manual intervention since (with the old system we would need to clean stray files from old builds now and then, but not anymore!) \o/
Only remaining queues on old EWS are windows, jsc and commit-queue, which I will be working on next.
Today Xan López and I talked a bit about the JSCOnly queue. We have a many builders targeting different architectures, which we would like to have moved to the new system. In the case of the JSC port, instead of grouping all of the builders under a single “jsc” status bubble it would be better to have a bubble for each target architecture e.g. “jsc-armv7”, “jsc-arm64”, and so on. The motivation is that we think it makes sense to that the system considers that a patch cannot be landed if it breaks any of the supported JSC architectures. With a single status bubble grouping them, we can have situations where a builder for one architecture passes the build+checks but the patch may still break *some other* architecture (e.g. a patch that touches ARM code generation gets picked by a MIPS builder). Is the above something that can supported? Let us know if you need more information and/or some support from our side for this. Best regards, —Adrián
On Sep 4, 2019, at 11:49 am, Adrian Perez de Castro <aperez@igalia.com> wrote:
Hello Aakash, WebKittens,
On Thu, 29 Aug 2019 18:01:56 -0400, Aakash Jain <aakash_jain@apple.com> wrote:
I want to update everyone with the further improvements I have made to new EWS. As always, please feel free to provide any feedback (either by filing bugs or contacting me directly).
New Features:
[...]
Thanks a ton for all the improvements in the EWS! Since we moved the GTK/EWS builders to the new queues they have been faster and more reliable. After solving a couple of minor hiccups right after the switch, the builders have not needed any manual intervention since (with the old system we would need to clean stray files from old builds now and then, but not anymore!)
Glad to hear. New EWS has been more reliable and easier to maintain at our end as well.
\o/
Only remaining queues on old EWS are windows, jsc and commit-queue, which I will be working on next.
Today Xan López and I talked a bit about the JSCOnly queue. We have a many builders targeting different architectures, which we would like to have moved to the new system.
In the case of the JSC port, instead of grouping all of the builders under a single “jsc” status bubble it would be better to have a bubble for each target architecture e.g. “jsc-armv7”, “jsc-arm64”, and so on.
Yes, my plan is to have separate bubbles for different architectures. I will touch base with you when I work on those queues.
The motivation is that we think it makes sense to that the system considers that a patch cannot be landed if it breaks any of the supported JSC architectures. With a single status bubble grouping them, we can have situations where a builder for one architecture passes the build+checks but the patch may still break *some other* architecture (e.g. a patch that touches ARM code generation gets picked by a MIPS builder).
Is the above something that can supported? Let us know if you need more information and/or some support from our side for this.
Best regards, —Adrián
participants (3)
-
Aakash Jain
-
Adrian Perez de Castro
-
Ryosuke Niwa