[webkit-dev] Huge improvement in Safari results on wpt.fyi

Philip J├Ągenstedt foolip at chromium.org
Tue Oct 2 03:45:26 PDT 2018


Hi WebKittens,

Fresh off the bots, I'm excited to report more robust Safari results,
and that Safari WPT pass rates are clearly improving! Thanks to the
hard work of Mike Pennisi [1] we now have the first Safari 12 results:
https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0

This uses the same setup as for Safari Technology Preview, which has
been running for a while [2] and are the results you see on the
"experimental" view:
https://wpt.fyi/results/?label=experimental

This appears much more robust than the Safari 11 data we've collected
from Sauce Labs, and we can see a massive improvement between Safari
11 and 12:
https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff

This lumps together infrastructure improvements as well as Safari
11->12 improvements, but improvements in service-workers/ [3] stands
out, as well as in webdriver/, referrer-policy/, css/css-align/, and
others. (The effect of moving away from Sauce is mainly less
timeouts.)

Also very interesting is to compare Safari 12 stable to TP:
https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1&diff

One can tell that work is going in canvas-related things,
web-animations/, css/css-logical/ and more! \o/

I hope you'll all find these results valuable, and please report bugs
or feature requests here:
https://github.com/web-platform-tests/wpt.fyi/issues

P.S. We're also trying to use use these diff views to spot
regressions. It's a bit hard to use, [4] but a fix in in progress [5]
and I might check back here when that works. I'll append to the end of
this email a non-exhaustive list of possible regressions already
possible to spot.

[1] https://github.com/web-platform-tests/results-collection/issues/604
[2] https://wpt.fyi/test-runs?labels=safari,experimental
[3] https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true
[4] https://github.com/web-platform-tests/wpt.fyi/issues/411
[5] https://github.com/web-platform-tests/wpt.fyi/pull/609

P.P.S. Possible regressions in Safari TP:
https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1


More information about the webkit-dev mailing list