[webkit-dev] Huge improvement in Safari results on wpt.fyi
dino at apple.com
Thu Oct 11 18:01:43 PDT 2018
It turns out that many (most?) of the CSS failures are because we no longer expose user-installed fonts, e.g. Ahem.
- update lots of tests to load Ahem via @font-face (yuck)
- allow Ahem to be used if installed (weird to special case one font, but probably ok)
> On 12 Oct 2018, at 03:26, Philip Jägenstedt <foolip at chromium.org> wrote:
> Alright, I've written a one-off script  to find the Safari-only
> failures, and here's the output:
> Barring bugs, each of linked tests or one of its subtests should be
> failing in Safari Technology Preview and passing in stable versions of
> Chrome, Edge and Firefox.
> Numerically, most of the failures are in css (622), encoding (135) and
> html (60). With css, it's mostly css/CSS2.
> I hope looking through this may be of use to you!
>  https://github.com/foolip/ad-hoc-wpt-results-analysis
> On Mon, Oct 8, 2018 at 11:50 PM Philip Jägenstedt <foolip at chromium.org> wrote:
>> That filtering capability unfortunately does not yet exist on wpt.fyi
>> but it's a high priority and actively being worked on:
>> FWIW, I suspect that these purposes, comparing to the stable versions
>> of all *other* browsers might be the most useful:
>> Again, no way to filter on wpt.fyi, but I'll see if I can download the
>> full results and write a quick script.
>> On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa <rniwa at webkit.org> wrote:
>>> Thanks for the intriguing data, Philip.
>>> Is there a way to get a list of tests where all other browsers pass but Safari / WebKit fail?
>>> That would allow us to quickly identify the set of tests we can fix to improve the interoperability across browsers right away.
>>> - R. Niwa
>>> On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt <foolip at chromium.org> wrote:
>>>> Hi WebKittens,
>>>> Fresh off the bots, I'm excited to report more robust Safari results,
>>>> and that Safari WPT pass rates are clearly improving! Thanks to the
>>>> hard work of Mike Pennisi  we now have the first Safari 12 results:
>>>> This uses the same setup as for Safari Technology Preview, which has
>>>> been running for a while  and are the results you see on the
>>>> "experimental" view:
>>>> This appears much more robust than the Safari 11 data we've collected
>>>> from Sauce Labs, and we can see a massive improvement between Safari
>>>> 11 and 12:
>>>> This lumps together infrastructure improvements as well as Safari
>>>> 11->12 improvements, but improvements in service-workers/  stands
>>>> out, as well as in webdriver/, referrer-policy/, css/css-align/, and
>>>> others. (The effect of moving away from Sauce is mainly less
>>>> Also very interesting is to compare Safari 12 stable to TP:
>>>> One can tell that work is going in canvas-related things,
>>>> web-animations/, css/css-logical/ and more! \o/
>>>> I hope you'll all find these results valuable, and please report bugs
>>>> or feature requests here:
>>>> P.S. We're also trying to use use these diff views to spot
>>>> regressions. It's a bit hard to use,  but a fix in in progress 
>>>> and I might check back here when that works. I'll append to the end of
>>>> this email a non-exhaustive list of possible regressions already
>>>> possible to spot.
>>>>  https://github.com/web-platform-tests/results-collection/issues/604
>>>>  https://wpt.fyi/test-runs?labels=safari,experimental
>>>>  https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true
>>>>  https://github.com/web-platform-tests/wpt.fyi/issues/411
>>>>  https://github.com/web-platform-tests/wpt.fyi/pull/609
>>>> P.P.S. Possible regressions in Safari TP:
>>>> webkit-dev mailing list
>>>> webkit-dev at lists.webkit.org
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
More information about the webkit-dev