[webkit-dev] Another WPT bite
youennf at gmail.com
Fri May 12 11:43:57 PDT 2017
Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov <ap at webkit.org> a écrit :
> 9 мая 2017 г., в 11:27, Simon Fraser <simon.fraser at apple.com> написал(а):
> Another consideration here is "would my test be useful for other browser
> vendors". I don't think the answer is a unanimous "yes", so I think we
> should only use WPT for tests that will think are worth sharing.
> Since imported WPT tests are very flaky, and are not necessarily written
> to defend against important regressions, investigating issues with them is
> relatively lower priority than investigating issues observed with WebKit
> tests. So I would recommend not mixing tests for WebKit regressions with
> WPT tests - if your test eventually ends up in LayoutTests/imported, it
> will become a lot less effective.
WPT tests are flaky in WebKit because WPT stability bots do not run yet
Safari, and most tests are written in non-WebKit environment.
Often, it is due to the fact that tests are not passing and flakiness is
only seen with failing assertions.
>From my experience with fetch API tests, I disagree that broken WPT tests
are lower priority.
I think it will change as more WebKit contributors will author WPT tests.
I agree that tests doing subtle WebKit-specific regression checks are not
good candidates for WPT upstream.
When the test is all about conformance with a spec, it is a very good
> Using the complicated harness has a similar consequence - if you use any
> WPT goodies like templates or server side scripts, the cost of
> investigating issues on the test increases, and makes the test less
It is true that WPT put some emphasis on easing authoring of tests.
I guess there is a learning curve here in WPT test debugging.
If you have a file with 20 tests, it is harder to debug.
It is also increasing the chances for flakiness/timeouts.
Maybe we could send that feedback.
WPT infra could also be improved: more verbose debug-only output, enabling
running selected subtest only...
testharness.js is actively maintained and is continuously improving.
Since we have specific requirements as you are describing, we should push
I don't know if there is any way to adopt WPT that won't make testing less
> effective. WPT tests may be useful in very rare cases, where we actively
> want to communicate certain subtle behavior details to other vendors - but
> even so, I imagine that other vendors may not put high priority on those,
> for the same reasons.
My own experience is that WPT tests are actually very valuable, at least
when we are talking about interoperability/spec conformance.
I also see WPT as an efficient way to author tests.
> - Alexey
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev