[webkit-dev] Huge improvement in Safari results on wpt.fyi

[Philip Jägenstedt]

Alright, I've written a one-off script [1] to find the Safari-only
failures, and here's the output:
https://gist.github.com/foolip/4d410ce79416bcdce71feb212159a02e

Barring bugs, each of linked tests or one of its subtests should be
failing in Safari Technology Preview and passing in stable versions of
Chrome, Edge and Firefox.

Numerically, most of the failures are in css (622), encoding (135) and
html (60). With css, it's mostly css/CSS2.

I hope looking through this may be of use to you!

[1] https://github.com/foolip/ad-hoc-wpt-results-analysis

On Mon, Oct 8, 2018 at 11:50 PM Philip Jägenstedt <foolip at chromium.org> wrote:
>
> That filtering capability unfortunately does not yet exist on wpt.fyi
> but it's a high priority and actively being worked on:
> https://github.com/web-platform-tests/wpt.fyi/issues/201
>
> FWIW, I suspect that these purposes, comparing to the stable versions
> of all *other* browsers might be the most useful:
> https://wpt.fyi/results/?product=chrome%5Bstable%5D&product=edge%5Bstable%5D&product=firefox%5Bstable%5D&product=safari%5Bexperimental%5D&aligned
>
> Again, no way to filter on wpt.fyi, but I'll see if I can download the
> full results and write a quick script.
>
> On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa <rniwa at webkit.org> wrote:
> >
> > Thanks for the intriguing data, Philip.
> >
> > Is there a way to get a list of tests where all other browsers pass but Safari / WebKit fail?
> >
> > That would allow us to quickly identify the set of tests we can fix to improve the interoperability across browsers right away.
> >
> > - R. Niwa
> >
> > On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt <foolip at chromium.org> wrote:
> >>
> >> Hi WebKittens,
> >>
> >> Fresh off the bots, I'm excited to report more robust Safari results,
> >> and that Safari WPT pass rates are clearly improving! Thanks to the
> >> hard work of Mike Pennisi [1] we now have the first Safari 12 results:
> >> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0
> >>
> >> This uses the same setup as for Safari Technology Preview, which has
> >> been running for a while [2] and are the results you see on the
> >> "experimental" view:
> >> https://wpt.fyi/results/?label=experimental
> >>
> >> This appears much more robust than the Safari 11 data we've collected
> >> from Sauce Labs, and we can see a massive improvement between Safari
> >> 11 and 12:
> >> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff
> >>
> >> This lumps together infrastructure improvements as well as Safari
> >> 11->12 improvements, but improvements in service-workers/ [3] stands
> >> out, as well as in webdriver/, referrer-policy/, css/css-align/, and
> >> others. (The effect of moving away from Sauce is mainly less
> >> timeouts.)
> >>
> >> Also very interesting is to compare Safari 12 stable to TP:
> >> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1&diff
> >>
> >> One can tell that work is going in canvas-related things,
> >> web-animations/, css/css-logical/ and more! \o/
> >>
> >> I hope you'll all find these results valuable, and please report bugs
> >> or feature requests here:
> >> https://github.com/web-platform-tests/wpt.fyi/issues
> >>
> >> P.S. We're also trying to use use these diff views to spot
> >> regressions. It's a bit hard to use, [4] but a fix in in progress [5]
> >> and I might check back here when that works. I'll append to the end of
> >> this email a non-exhaustive list of possible regressions already
> >> possible to spot.
> >>
> >> [1] https://github.com/web-platform-tests/results-collection/issues/604
> >> [2] https://wpt.fyi/test-runs?labels=safari,experimental
> >> [3] https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true
> >> [4] https://github.com/web-platform-tests/wpt.fyi/issues/411
> >> [5] https://github.com/web-platform-tests/wpt.fyi/pull/609
> >>
> >> P.P.S. Possible regressions in Safari TP:
> >> https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
> >> https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
> >> https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
> >> _______________________________________________
> >> webkit-dev mailing list
> >> webkit-dev at lists.webkit.org
> >> https://lists.webkit.org/mailman/listinfo/webkit-dev