[webkit-dev] Filtering results on wpt.fyi, Safari-specific failures

[Philip Jägenstedt]

I'd like to point out right away that diagnosing reftest failures is
currently cumbersome because we don't store the screenshots. This is
also a work in progress:
https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing

Until that has launched, I would recommend ignoring reftest failures
if the cause of failure isn't obvious.

On Mon, Feb 25, 2019 at 2:30 PM Philip Jägenstedt <foolip at chromium.org> wrote:
>
> Hi all,
>
> Following the improved Safari results last year [1] and the discussion
> that generated, I'm happy to announce that the filtering requested as
> now available in the search box. The full syntax is documented [2] but
> there's also a new insights view [3] with some useful searches.
>
> Especially interesting for this list could be this view, of Chrome
> Dev, Firefox Nightly and Safari Technology Preview, filtered to the
> Safari-specific failures:
> https://wpt.fyi/results/?label=master&label=experimental&product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29
>
> Both Google and Mozilla have efforts [4][5] to reduce the number of
> Chrome/Firefox-specific failures, as this seems like a category of
> problems which especially valuable, where changing just one browser
> can remove a pain point for web developers.
>
> No doubt some failures are spurious, but hopefully there is value to
> be found by looking into where the largest numbers of failures appear
> to be. If something seems to be wrong with the search/filtering,
> please file an issue for us! [6]
>
> Credit to Mark Dittmer and Luke Bjerring who owned this project.
>
> P.S. We are also working on triage metadata for wpt.fyi, to make it
> possible to burn down a list of failures like this and not later have
> to re-triage to find the new failures. [7]
>
> [1] https://lists.webkit.org/pipermail/webkit-dev/2018-October/030209.html
> [2] https://github.com/web-platform-tests/wpt.fyi/blob/master/api/query/README.md
> [3] https://staging.wpt.fyi/insights
> [4] https://bugs.chromium.org/p/chromium/issues/detail?id=896242
> [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1498357
> [6] https://github.com/web-platform-tests/wpt.fyi/issues/new?title=Structured+Queries+issue&projects=web-platform-tests/wpt.fyi/8&labels=bug&template=search.md
> [7] https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit?usp=sharing