[webkit-dev] Huge improvement in Safari results on wpt.fyi

Philip J├Ągenstedt foolip at chromium.org
Mon Oct 8 14:50:25 PDT 2018

That filtering capability unfortunately does not yet exist on wpt.fyi
but it's a high priority and actively being worked on:

FWIW, I suspect that these purposes, comparing to the stable versions
of all *other* browsers might be the most useful:

Again, no way to filter on wpt.fyi, but I'll see if I can download the
full results and write a quick script.

On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa <rniwa at webkit.org> wrote:
> Thanks for the intriguing data, Philip.
> Is there a way to get a list of tests where all other browsers pass but Safari / WebKit fail?
> That would allow us to quickly identify the set of tests we can fix to improve the interoperability across browsers right away.
> - R. Niwa
> On Tue, Oct 2, 2018 at 3:45 AM Philip J├Ągenstedt <foolip at chromium.org> wrote:
>> Hi WebKittens,
>> Fresh off the bots, I'm excited to report more robust Safari results,
>> and that Safari WPT pass rates are clearly improving! Thanks to the
>> hard work of Mike Pennisi [1] we now have the first Safari 12 results:
>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0
>> This uses the same setup as for Safari Technology Preview, which has
>> been running for a while [2] and are the results you see on the
>> "experimental" view:
>> https://wpt.fyi/results/?label=experimental
>> This appears much more robust than the Safari 11 data we've collected
>> from Sauce Labs, and we can see a massive improvement between Safari
>> 11 and 12:
>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff
>> This lumps together infrastructure improvements as well as Safari
>> 11->12 improvements, but improvements in service-workers/ [3] stands
>> out, as well as in webdriver/, referrer-policy/, css/css-align/, and
>> others. (The effect of moving away from Sauce is mainly less
>> timeouts.)
>> Also very interesting is to compare Safari 12 stable to TP:
>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1&diff
>> One can tell that work is going in canvas-related things,
>> web-animations/, css/css-logical/ and more! \o/
>> I hope you'll all find these results valuable, and please report bugs
>> or feature requests here:
>> https://github.com/web-platform-tests/wpt.fyi/issues
>> P.S. We're also trying to use use these diff views to spot
>> regressions. It's a bit hard to use, [4] but a fix in in progress [5]
>> and I might check back here when that works. I'll append to the end of
>> this email a non-exhaustive list of possible regressions already
>> possible to spot.
>> [1] https://github.com/web-platform-tests/results-collection/issues/604
>> [2] https://wpt.fyi/test-runs?labels=safari,experimental
>> [3] https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true
>> [4] https://github.com/web-platform-tests/wpt.fyi/issues/411
>> [5] https://github.com/web-platform-tests/wpt.fyi/pull/609
>> P.P.S. Possible regressions in Safari TP:
>> https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
>> https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
>> https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev

More information about the webkit-dev mailing list