[webkit-dev] New EWS queue: Stress Test EWS

[Aakash Jain]

Hi Everyone,

I am happy to announce that I have added a new EWS queue: Stress Test EWS. From now on, you would see a new status bubbles (on Bugzilla bug) named: 'mac-wk2-stress'. For the patches that add or modify layout tests, this EWS will run those added/modified layout-tests repeatedly large number of times (currently 100 iterations). It will also run those added/modified layout-tests repeatedly in guard-malloc mode. Example: https://ews-build.webkit.org/#/builders/62/builds/249. The aim is to find any issue with the layout-tests, especially flakiness, or failure when run in a certain manner.

Flaky layout-tests are currently a huge problem, as they sometimes cause false positives on EWS (and other parts of infrastructure), and slow down our infrastructure (as for every test failure, EWS needs to run the test-suite few times to ensure the failures are not flaky or pre-existing). Until now, we didn't have any good automated way to indicate the test flakiness to the engineers. With this EWS, we are adding pre-commit testing for layout-test flakiness. We understand that layout tests might be flaky in a variety of ways, sometimes in a very specific or unexpected manner. I am looking at covering more and more scenarios in this EWS by making incremental improvements to it.

Few ideas to further improve this EWS are:
- Test on more OS/configurations (this EWS is currently using macOS release wk2 mode)
- Run the test being added/modified repeatedly along-with other layout-tests in same directory
- Improve the logic to identify layout-tests, current logic is naive and only recognizes html layout-tests

More ideas are welcome.

Thanks
Aakash