Hi Everyone, I am happy to announce that I have added a new EWS queue: Stress Test EWS. From now on, you would see a new status bubbles (on Bugzilla bug) named: 'mac-wk2-stress'. For the patches that add or modify layout tests, this EWS will run those added/modified layout-tests repeatedly large number of times (currently 100 iterations). It will also run those added/modified layout-tests repeatedly in guard-malloc mode. Example: https://ews-build.webkit.org/#/builders/62/builds/249. The aim is to find any issue with the layout-tests, especially flakiness, or failure when run in a certain manner. Flaky layout-tests are currently a huge problem, as they sometimes cause false positives on EWS (and other parts of infrastructure), and slow down our infrastructure (as for every test failure, EWS needs to run the test-suite few times to ensure the failures are not flaky or pre-existing). Until now, we didn't have any good automated way to indicate the test flakiness to the engineers. With this EWS, we are adding pre-commit testing for layout-test flakiness. We understand that layout tests might be flaky in a variety of ways, sometimes in a very specific or unexpected manner. I am looking at covering more and more scenarios in this EWS by making incremental improvements to it. Few ideas to further improve this EWS are: - Test on more OS/configurations (this EWS is currently using macOS release wk2 mode) - Run the test being added/modified repeatedly along-with other layout-tests in same directory - Improve the logic to identify layout-tests, current logic is naive and only recognizes html layout-tests More ideas are welcome. Thanks Aakash
Hi, The stress test EWS has some issue when dealing with testharness.js tests. Every now and then it thinks it's a different type of test and it dumps the layout tree, and it fails as the actual result has nothing to do with a layout tree dump. Actually it dumps an empty layout tree: layer at (0,0) size 800x600 RenderView at (0,0) size 800x600 layer at (0,0) size 800x600 RenderBlock {HTML} at (0,0) size 800x600 RenderBody {BODY} at (8,8) size 784x584 Last example I've seen: https://ews-build.webkit.org/#/builders/62/builds/1903 I guess there might be some timing issue or something like that, as it looks like it doesn't even load the test properly before comparing the results. Cheers, Rego
Hi, This issue is tracked as https://bugs.webkit.org/show_bug.cgi?id=225126 <https://bugs.webkit.org/show_bug.cgi?id=225126> - Alexey
28 апр. 2021 г., в 3:23 AM, Manuel Rego Casasnovas via webkit-dev <webkit-dev@lists.webkit.org> написал(а):
Hi,
The stress test EWS has some issue when dealing with testharness.js tests.
Every now and then it thinks it's a different type of test and it dumps the layout tree, and it fails as the actual result has nothing to do with a layout tree dump.
Actually it dumps an empty layout tree: layer at (0,0) size 800x600 RenderView at (0,0) size 800x600 layer at (0,0) size 800x600 RenderBlock {HTML} at (0,0) size 800x600 RenderBody {BODY} at (8,8) size 784x584
Last example I've seen: https://ews-build.webkit.org/#/builders/62/builds/1903
I guess there might be some timing issue or something like that, as it looks like it doesn't even load the test properly before comparing the results.
Cheers, Rego _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
participants (3)
-
Aakash Jain
-
Alexey Proskuryakov
-
Manuel Rego Casasnovas