[webkit-dev] Another WPT bite

[Rick Byers]

On Fri, May 12, 2017 at 2:43 PM, youenn fablet <youennf at gmail.com> wrote:

>
> Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov <ap at webkit.org> a écrit :
>
>>
>> 9 мая 2017 г., в 11:27, Simon Fraser <simon.fraser at apple.com> написал(а):
>>
>>
>> Another consideration here is "would my test be useful for other browser
>> vendors". I don't think the answer is a unanimous "yes", so I think we
>> should only use WPT for tests that will think are worth sharing.
>>
>>
>> Since imported WPT tests are very flaky, and are not necessarily written
>> to defend against important regressions, investigating issues with them is
>> relatively lower priority than investigating issues observed with WebKit
>> tests. So I would recommend not mixing tests for WebKit regressions with
>> WPT tests - if your test eventually ends up in LayoutTests/imported, it
>> will become a lot less effective.
>>
>
> WPT tests are flaky in WebKit because WPT stability bots do not run yet
> Safari, and most tests are written in non-WebKit environment.
> Often, it is due to the fact that tests are not passing and flakiness is
> only seen with failing assertions.
>
> From my experience with fetch API tests, I disagree that broken WPT tests
> are lower priority.
> I think it will change as more WebKit contributors will author WPT tests.
>
> I agree that tests doing subtle WebKit-specific regression checks are not
> good candidates for WPT upstream.
> When the test is all about conformance with a spec, it is a very good
> candidate.
>
>
>> Using the complicated harness has a similar consequence - if you use any
>> WPT goodies like templates or server side scripts, the cost of
>> investigating issues on the test increases, and makes the test less
>> valuable.
>>
>
> It is true that WPT put some emphasis on easing authoring of tests.
> I guess there is a learning curve here in WPT test debugging.
>
> If you have a file with 20 tests, it is harder to debug.
> It is also increasing the chances for flakiness/timeouts.
> Maybe we could send that feedback.
>
> WPT infra could also be improved: more verbose debug-only output, enabling
> running selected subtest only...
> testharness.js is actively maintained and is continuously improving.
> Since we have specific requirements as you are describing, we should push
> them.
>

Concretely, please file issues with the "infra" label
<https://github.com/w3c/web-platform-tests/issues?q=is%3Aissue+is%3Aopen+label%3Ainfra>
to
track upstream WPT infrastructure bugs and feature requests.  Google has an
ongoing contract with Bocoup (Bob and Mike cc'd) who are improving the
infrastructure every day.  Of course there's an opportunity to make it even
better fast with more funding from additional organizations who would
benefit :-)

>
> I don't know if there is any way to adopt WPT that won't make testing less
>> effective. WPT tests may be useful in very rare cases, where we actively
>> want to communicate certain subtle behavior details to other vendors - but
>> even so, I imagine that other vendors may not put high priority on those,
>> for the same reasons.
>>
>
> My own experience is that WPT tests are actually very valuable, at least
> when we are talking about interoperability/spec conformance.
> I also see WPT as an efficient way to author tests.
>
>
>>
>> - Alexey
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170512/4d2cec9f/attachment.html>