[webkit-dev] Filtering results on wpt.fyi, Safari-specific failures

Philip J├Ągenstedt foolip at chromium.org
Tue Apr 2 04:15:07 PDT 2019

On Fri, Mar 29, 2019 at 6:16 PM Robert Ma <robertma at chromium.org> wrote:

> On Mon, Feb 25, 2019 at 8:49 AM Philip J├Ągenstedt <foolip at chromium.org>
> wrote:
>> I'd like to point out right away that diagnosing reftest failures is
>> currently cumbersome because we don't store the screenshots. This is
>> also a work in progress:
>> https://docs.google.com/document/d/1IhZa4mrjK1msUMhtamKwKJ_HhXD-nqh_4-BcPWM6soQ/edit?usp=sharing
>> Until that has launched, I would recommend ignoring reftest failures
>> if the cause of failure isn't obvious.
> Great news! Reftest screenshots are now available on wpt.fyi. No more
> guesswork for why a reftest fails!
> For example, this
> <https://wpt.fyi/results/css/css-flexbox/flex-wrap-002.html?label=master&label=experimental&product=chrome%5Btaskcluster%5D&product=firefox%5Btaskcluster%5D&product=safari%5Bazure%5D&aligned&q=%28chrome%3Apass%7Cchrome%3Aok%29+%28firefox%3Apass%7Cfirefox%3Aok%29+%28safari%3A%21pass%26safari%3A%21ok%29>
> is one of the Safari-only reftest failures you can find using the search
> link posted earlier. Now you can click the "compare" button (you might need
> to force-reload the page to see it) to view the screenshots. This example
> looks like a genuine failure, while some others are probably caused by font
> antialiasing/kerning (they should most likely use the Ahem font instead).
> We are also working on another feature to triage the failures
> <https://docs.google.com/document/d/1oWYVkc2ztANCGUxwNVTQHlWV32zq6Ifq9jkkbYNbSAg/edit>
> (e.g. to mark a test as a genuine failure and link it to bug trackers, or
> as flaky/broken). Stay tuned!

The screenshots can also come in handy when comparing Safari stable to
Technology Preview:

/css/css-contain/contain-layout-baseline-003.html is one reftest that
appears to have regressed in Technology Preview, and one can see the
failure here:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20190402/8269783a/attachment.html>

More information about the webkit-dev mailing list