[webkit-dev] Safari Tech Preview 96 available on wpt.fyi!

[Stephen Mcgruer]

>
> There’s a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
> Many large new chunks of not-run tests are caused by a harness error or
> timeout. E.g. html/ and webauthn


We have unfortunately struggled historically with getting reliable +
consistent results for Safari, with spurious infrastructure + other
problems. That said, Igalia has been working on improving reliability and
have made good progress, and we have also invested in projects like the
WebkitGTK runs to help with this problem (e.g. if a test fails in both
Safari and WebKitGTK, there's a better chance it's a real failure[0])

We have had many Safari TP 96 runs since my announcement (
https://wpt.fyi/runs?label=master&max-count=100&product=safari); you can
click on a single run (the icon) to view the run itself, or click on two
runs to get a diff view - which can be useful for spotting flakes (as most
SHAs are only changing a few tests, so differences between consecutive runs
are usually flakes). There is also an explicit flake analyzer on our
insights page - https://wpt.fyi/insights (it's a little clunky but can
still be useful).

I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.


Improvements would be great! Looking at that PR, it is quite old, so will
need to be rebased and force pushed. Let me know if you need any help on
that :).

STP 96 has enabled user agent UI for WebAuthn, which could cause crashes.
> This could be the reason why webauthn is 0 all the time.


It does look like there are still some harness failures in further STP 96
runs (e.g. https://wpt.fyi/results/webauthn?run_id=361070008); I see a
mention in the failing tests of a "UnexpectedAlertOpenException: unexpected
alert open (500):" - is that what you mean? Is there some flag we could be
setting on STP, or something Safaridriver could be doing to avoid these?

[0]:
https://wpt.fyi/results/?label=master&product=safari%5Bexperimental%5D&product=webkitgtk&product=chrome%5Bexperimental%5D&product=firefox%5Bexperimental%5D&aligned&q=safari%3Afail%20webkitgtk%3Afail%20chrome%3Apass%20firefox%3Apass
is
an example search for tests that fail in both Safari + WebKitGTK, but pass
in both Chrome and Firefox.

On Mon, 2 Dec 2019 at 21:04, Jiewen Tan <jiewen_tan at apple.com> wrote:

> Hi Maciej,
>
> On Dec 2, 2019, at 4:10 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>
> There’s a number of mysterious timeouts with 96. Not sure if flakiness or
> real?
>
> The new WebCrypto failures are surprising, but likely real and should be
> investigated:
> https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013
> <https://wpt.fyi/results/WebCryptoAPI/wrapKey_unwrapKey?diff&filter=ADC&q=is:different&run_id=347530011&run_id=381930013>
>
>
> I believe these tests are flaky. I have made a PR to improve it a while
> ago. I should probably get those improvement landed sometime.
> https://github.com/web-platform-tests/wpt/pull/6102
>
>
> Many large new chunks of not-run tests are caused by a harness error or
> timeout. E.g. html/ and webauthn/
>
>
> STP 96 has enabled user agent UI for WebAuthn, which could cause crashes.
> This could be the reason why webauthn is 0 all the time.
>
> Best,
> Jiewen
>
>
>
> On Nov 27, 2019, at 7:07 AM, Stephen Mcgruer <smcgruer at chromium.org>
> wrote:
>
> Excited to announce that Safari Tech Preview 96 is now available on
> wpt.fyi!
>
> Example run:
> https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned
>
> Diff against Safari Tech Preview 95 (not at exactly the same WPT sha, but
> very close):
> https://wpt.fyi/results/?diff&filter=ADC&q=is%3Adifferent&run_id=347530011&run_id=381930013
>
> If you are surprised by the results in the diff view, I would love to hear
> about it. One thing that surprised me is that the release blog post[0]
> mentioned Web Animations being enabled by default, but we see no
> differences in the test diff for web-animations/. It's possible we enable
> some flag that turned on Web Animations already, but I can't see it
> obviously in our safari setup[1].
>
> [0]:
> https://webkit.org/blog/9658/release-notes-for-safari-technology-preview-96/
> [1]:
> https://github.com/web-platform-tests/wpt/blob/master/tools/ci/azure/install_safari.yml
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20191202/1db57c80/attachment.htm>