[Webkit-unassigned] [Bug 248561] New: [EWS] Include flavor (wk1/wk2) in results database request when evaluating pre-existing failures

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=248561

            Bug ID: 248561
           Summary: [EWS] Include flavor (wk1/wk2) in results database
                    request when evaluating pre-existing failures
           Product: WebKit
           Version: Other
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: ryanhaddad at apple.com

The test failures seen in the following WK1 run were consistently failing on ToT WK1 until they were rebaselined last night, but the failures were still triggering retries in spite of our new results database checks.

https://ews-build.webkit.org/#/builders/73/builds/20801

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcss%2Fcss-cascade%2Fall-prop-initial-xml.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fgeolocation-API%2FPositionOptions.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fgeolocation-API%2Fpermission.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Faccumulation-per-property-002.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Faddition-per-property-002.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Finterpolation-per-property-002.html


The issue might be that when we query the results database to analyze failures, we are only checking for platform (macOS/iOS) and configuration (release/debug), which will return results from both WK1 and WK2 and throw off the pass/fail percentage.

Checking Results database for failing tests. Identifier: 257097 at main, configuration: {'platform': 'mac', 'style': 'release'}
imported/w3c/web-platform-tests/css/css-cascade/all-prop-initial-xml.html: pass_rate: 72, pre-existing-failure=False
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 72, 'text': 27, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/geolocation-API/PositionOptions.https.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/geolocation-API/permission.https.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/accumulation-per-property-002.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/addition-per-property-002.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}
imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/interpolation-per-property-002.html: pass_rate: 44, pre-existing-failure=True
Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0}


If we had been checking for WK1 results specifically, it seems likely that we would have ignored the failures and skipped the retry step.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20221130/bbee2add/attachment.htm>