[webkit-dev] Another WPT bite

[youenn fablet]

Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov <ap at webkit.org> a écrit :

>
> 9 мая 2017 г., в 11:27, Simon Fraser <simon.fraser at apple.com> написал(а):
>
>
> Another consideration here is "would my test be useful for other browser
> vendors". I don't think the answer is a unanimous "yes", so I think we
> should only use WPT for tests that will think are worth sharing.
>
>
> Since imported WPT tests are very flaky, and are not necessarily written
> to defend against important regressions, investigating issues with them is
> relatively lower priority than investigating issues observed with WebKit
> tests. So I would recommend not mixing tests for WebKit regressions with
> WPT tests - if your test eventually ends up in LayoutTests/imported, it
> will become a lot less effective.
>

WPT tests are flaky in WebKit because WPT stability bots do not run yet
Safari, and most tests are written in non-WebKit environment.
Often, it is due to the fact that tests are not passing and flakiness is
only seen with failing assertions.

>From my experience with fetch API tests, I disagree that broken WPT tests
are lower priority.
I think it will change as more WebKit contributors will author WPT tests.

I agree that tests doing subtle WebKit-specific regression checks are not
good candidates for WPT upstream.
When the test is all about conformance with a spec, it is a very good
candidate.


> Using the complicated harness has a similar consequence - if you use any
> WPT goodies like templates or server side scripts, the cost of
> investigating issues on the test increases, and makes the test less
> valuable.
>

It is true that WPT put some emphasis on easing authoring of tests.
I guess there is a learning curve here in WPT test debugging.

If you have a file with 20 tests, it is harder to debug.
It is also increasing the chances for flakiness/timeouts.
Maybe we could send that feedback.

WPT infra could also be improved: more verbose debug-only output, enabling
running selected subtest only...
testharness.js is actively maintained and is continuously improving.
Since we have specific requirements as you are describing, we should push
them.

I don't know if there is any way to adopt WPT that won't make testing less
> effective. WPT tests may be useful in very rare cases, where we actively
> want to communicate certain subtle behavior details to other vendors - but
> even so, I imagine that other vendors may not put high priority on those,
> for the same reasons.
>

My own experience is that WPT tests are actually very valuable, at least
when we are talking about interoperability/spec conformance.
I also see WPT as an efficient way to author tests.


>
> - Alexey
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170512/489ec223/attachment-0001.html>