Neat.

I see some obvious areas for focus, where Safari fails lots of tests that the other browser don’t. 

For context, I tried looking at this view, which shows all tests that Safari and Firefox pass with Safari results regardless of result:

https://wpt.fyi/results/?label=master&label=experimental&product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29

I noticed some puzzling results there: Safari passes all the ambient-light and bluetooth tests that Chrome and Firefox do, despite not supporting these standards at all. (For that matter I’m not sure Firefox supports these specs either.) Not sure if harness problem, or dubious tests that don’t actually test the standard.

Regards,

Maciej

On Feb 25, 2019, at 5:48 AM, Philip Jägenstedt <foolip@chromium.org> wrote:

I'd like to point out right away that diagnosing reftest failures is
currently cumbersome because we don't store the screenshots. This is
also a work in progress:
https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing

Until that has launched, I would recommend ignoring reftest failures
if the cause of failure isn't obvious.

On Mon, Feb 25, 2019 at 2:30 PM Philip Jägenstedt <foolip@chromium.org> wrote:

Hi all,

Following the improved Safari results last year [1] and the discussion
that generated, I'm happy to announce that the filtering requested as
now available in the search box. The full syntax is documented [2] but
there's also a new insights view [3] with some useful searches.

Especially interesting for this list could be this view, of Chrome
Dev, Firefox Nightly and Safari Technology Preview, filtered to the
Safari-specific failures:
https://wpt.fyi/results/?label=master&label=experimental&product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29

Both Google and Mozilla have efforts [4][5] to reduce the number of
Chrome/Firefox-specific failures, as this seems like a category of
problems which especially valuable, where changing just one browser
can remove a pain point for web developers.

No doubt some failures are spurious, but hopefully there is value to
be found by looking into where the largest numbers of failures appear
to be. If something seems to be wrong with the search/filtering,
please file an issue for us! [6]

Credit to Mark Dittmer and Luke Bjerring who owned this project.

P.S. We are also working on triage metadata for wpt.fyi, to make it
possible to burn down a list of failures like this and not later have
to re-triage to find the new failures. [7]

[1] https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html
[2] https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md
[3] https://staging.wpt.fyi/insights
[4] https://bugs.chromium.org/p/chromium/issues/detail?id=896242
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1498357
[6] https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue&projects=web-platform-tests/wpt.fyi/8&labels=bug&template=search.md
[7] https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing

_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
https://lists.webkit.org/mailman/listinfo/webkit-dev