[webkit-dev] Huge improvement in Safari results on wpt.fyi

Thu Oct 11 20:22:52 PDT 2018

On 10/12/18 3:59 AM, Geoffrey Garen wrote:
> Honest question: What’s gross about using @font-face?
> 
> It would be lots of test edits. That’s a bummer.
> 
> But maybe it’s clearer for the tests to specify the font they want to use. It makes the test self-describing, eliminating the requirement that the user take a step outside the test to get the right result.

Note that there's also the opposite opinion of loading a web font 
potentially hiding bugs:

   https://lists.w3.org/Archives/Public/www-style/2017Jan/0053.html

Though I don't have such a strong opinion myself, I think @font-face is 
a fine solution for that problem (and other people seemed to be ok with 
that as well, looking at how that thread continues...).

I don't know if the CSSWG ended up taking an official position on this, 
but may be worth asking in www-style before doing he work of a mass-convert.

  -- Emilio

> Thanks,
> Geoff
> 
>> On Oct 11, 2018, at 6:01 PM, Dean Jackson <dino at apple.com> wrote:
>>
>> It turns out that many (most?) of the CSS failures are because we no longer expose user-installed fonts, e.g. Ahem.
>>
>> Options:
>>
>> - update lots of tests to load Ahem via @font-face (yuck)
>> - allow Ahem to be used if installed (weird to special case one font, but probably ok)
>>
>> Dean
>>
>>> On 12 Oct 2018, at 03:26, Philip Jägenstedt <foolip at chromium.org> wrote:
>>>
>>> Alright, I've written a one-off script [1] to find the Safari-only
>>> failures, and here's the output:
>>> https://gist.github.com/foolip/4d410ce79416bcdce71feb212159a02e
>>>
>>> Barring bugs, each of linked tests or one of its subtests should be
>>> failing in Safari Technology Preview and passing in stable versions of
>>> Chrome, Edge and Firefox.
>>>
>>> Numerically, most of the failures are in css (622), encoding (135) and
>>> html (60). With css, it's mostly css/CSS2.
>>>
>>> I hope looking through this may be of use to you!
>>>
>>> [1] https://github.com/foolip/ad-hoc-wpt-results-analysis
>>>
>>> On Mon, Oct 8, 2018 at 11:50 PM Philip Jägenstedt <foolip at chromium.org> wrote:
>>>>
>>>> That filtering capability unfortunately does not yet exist on wpt.fyi
>>>> but it's a high priority and actively being worked on:
>>>> https://github.com/web-platform-tests/wpt.fyi/issues/201
>>>>
>>>> FWIW, I suspect that these purposes, comparing to the stable versions
>>>> of all *other* browsers might be the most useful:
>>>> https://wpt.fyi/results/?product=chrome%5Bstable%5D&product=edge%5Bstable%5D&product=firefox%5Bstable%5D&product=safari%5Bexperimental%5D&aligned
>>>>
>>>> Again, no way to filter on wpt.fyi, but I'll see if I can download the
>>>> full results and write a quick script.
>>>>
>>>> On Thu, Oct 4, 2018 at 11:49 PM Ryosuke Niwa <rniwa at webkit.org> wrote:
>>>>>
>>>>> Thanks for the intriguing data, Philip.
>>>>>
>>>>> Is there a way to get a list of tests where all other browsers pass but Safari / WebKit fail?
>>>>>
>>>>> That would allow us to quickly identify the set of tests we can fix to improve the interoperability across browsers right away.
>>>>>
>>>>> - R. Niwa
>>>>>
>>>>> On Tue, Oct 2, 2018 at 3:45 AM Philip Jägenstedt <foolip at chromium.org> wrote:
>>>>>>
>>>>>> Hi WebKittens,
>>>>>>
>>>>>> Fresh off the bots, I'm excited to report more robust Safari results,
>>>>>> and that Safari WPT pass rates are clearly improving! Thanks to the
>>>>>> hard work of Mike Pennisi [1] we now have the first Safari 12 results:
>>>>>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0
>>>>>>
>>>>>> This uses the same setup as for Safari Technology Preview, which has
>>>>>> been running for a while [2] and are the results you see on the
>>>>>> "experimental" view:
>>>>>> https://wpt.fyi/results/?label=experimental
>>>>>>
>>>>>> This appears much more robust than the Safari 11 data we've collected
>>>>>> from Sauce Labs, and we can see a massive improvement between Safari
>>>>>> 11 and 12:
>>>>>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff
>>>>>>
>>>>>> This lumps together infrastructure improvements as well as Safari
>>>>>> 11->12 improvements, but improvements in service-workers/ [3] stands
>>>>>> out, as well as in webdriver/, referrer-policy/, css/css-align/, and
>>>>>> others. (The effect of moving away from Sauce is mainly less
>>>>>> timeouts.)
>>>>>>
>>>>>> Also very interesting is to compare Safari 12 stable to TP:
>>>>>> https://wpt.fyi/results/?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1&diff
>>>>>>
>>>>>> One can tell that work is going in canvas-related things,
>>>>>> web-animations/, css/css-logical/ and more! \o/
>>>>>>
>>>>>> I hope you'll all find these results valuable, and please report bugs
>>>>>> or feature requests here:
>>>>>> https://github.com/web-platform-tests/wpt.fyi/issues
>>>>>>
>>>>>> P.S. We're also trying to use use these diff views to spot
>>>>>> regressions. It's a bit hard to use, [4] but a fix in in progress [5]
>>>>>> and I might check back here when that works. I'll append to the end of
>>>>>> this email a non-exhaustive list of possible regressions already
>>>>>> possible to spot.
>>>>>>
>>>>>> [1] https://github.com/web-platform-tests/results-collection/issues/604
>>>>>> [2] https://wpt.fyi/test-runs?labels=safari,experimental
>>>>>> [3] https://wpt.fyi/results/service-workers?sha=ee2e69bfb1&product=safari-11.1&product=safari-12.0&diff=true
>>>>>> [4] https://github.com/web-platform-tests/wpt.fyi/issues/411
>>>>>> [5] https://github.com/web-platform-tests/wpt.fyi/pull/609
>>>>>>
>>>>>> P.P.S. Possible regressions in Safari TP:
>>>>>> https://wpt.fyi/results/css/vendor-imports/mozilla/mozilla-central-reftests/shapes1?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
>>>>>> https://wpt.fyi/results/service-workers/service-worker/extendable-event-async-waituntil.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
>>>>>> https://wpt.fyi/results/service-workers/service-worker/skip-waiting-installed.https.html?sha=ee2e69bfb1&product=safari-12.0&product=safari-12.1
>>>>>> _______________________________________________
>>>>>> webkit-dev mailing list
>>>>>> webkit-dev at lists.webkit.org
>>>>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
> 