[Webkit-unassigned] [Bug 212381] New: [EWS] Add a special case for running the layout test step without aborting in case of many failures for WPT tests

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=212381

            Bug ID: 212381
           Summary: [EWS] Add a special case for running the layout test
                    step without aborting in case of many failures for WPT
                    tests
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: clopez at igalia.com
                CC: aakash_jain at apple.com, ap at webkit.org,
                    jbedard at apple.com, youennf at gmail.com

I want to prototype a bot that helps to update weekly our WPT import checkout <https://lists.webkit.org/pipermail/webkit-dev/2020-March/031121.html>

And for doing this, I want to (re-)use our EWS system to get the layout test results for all the platforms (Mac, iOS, etc..).

However, I'm finding that the EWS layout test results are not very useful when trying patches that touch lot of tests (like a WPT import)

This is due to the EWS running the layout test step with:

 * "--exit-after-n-failures 30" -> usually a WPT import are thousands of new tests, if it aborts instead of giving back all the failures it is not really useful
 * "--skip-failing-tests" -> some tests may crash (or timeout) instead of fail in different platforms, if we don't try to re-recheck if they fail we miss that. See for example r262062 (I only found that after landing the patch, because I had marked that test as failing (since it fails on release), but the EWS debug was skipping it, therefore I couldn't know it was crashing on debug)

I understand this parameters are useful to make the EWS faster and are good enough for the general use-case

So I want to propose something like not passing those parameters to the EWS for patches uploaded by a specific bugzilla user (the one this bot would use).
That way the EWS can be useful for this bot, meanwhile we keep the current behavior for everybody else.

Also it will help (to make everything faster and simplify everything) that for this user the layout test step only runs the tests inside the folder "imported/w3c/web-platform-tests/", since those are the only ones interesting for it anyway.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20200526/d1e2aefb/attachment.htm>