[webkit-dev] Another WPT bite

[youenn fablet]

Filed https://github.com/w3c/web-platform-tests/issues/5909 and
https://github.com/w3c/web-platform-tests/issues/5910.


Le ven. 12 mai 2017 à 12:08, Rick Byers <rbyers at chromium.org> a écrit :

> On Fri, May 12, 2017 at 2:43 PM, youenn fablet <youennf at gmail.com> wrote:
>
>>
>> Le ven. 12 mai 2017 à 11:07, Alexey Proskuryakov <ap at webkit.org> a
>> écrit :
>>
>>>
>>> 9 мая 2017 г., в 11:27, Simon Fraser <simon.fraser at apple.com>
>>> написал(а):
>>>
>>>
>>> Another consideration here is "would my test be useful for other browser
>>> vendors". I don't think the answer is a unanimous "yes", so I think we
>>> should only use WPT for tests that will think are worth sharing.
>>>
>>>
>>> Since imported WPT tests are very flaky, and are not necessarily written
>>> to defend against important regressions, investigating issues with them is
>>> relatively lower priority than investigating issues observed with WebKit
>>> tests. So I would recommend not mixing tests for WebKit regressions with
>>> WPT tests - if your test eventually ends up in LayoutTests/imported, it
>>> will become a lot less effective.
>>>
>>
>> WPT tests are flaky in WebKit because WPT stability bots do not run yet
>> Safari, and most tests are written in non-WebKit environment.
>> Often, it is due to the fact that tests are not passing and flakiness is
>> only seen with failing assertions.
>>
>> From my experience with fetch API tests, I disagree that broken WPT tests
>> are lower priority.
>> I think it will change as more WebKit contributors will author WPT tests.
>>
>> I agree that tests doing subtle WebKit-specific regression checks are not
>> good candidates for WPT upstream.
>> When the test is all about conformance with a spec, it is a very good
>> candidate.
>>
>>
>>> Using the complicated harness has a similar consequence - if you use any
>>> WPT goodies like templates or server side scripts, the cost of
>>> investigating issues on the test increases, and makes the test less
>>> valuable.
>>>
>>
>> It is true that WPT put some emphasis on easing authoring of tests.
>> I guess there is a learning curve here in WPT test debugging.
>>
>> If you have a file with 20 tests, it is harder to debug.
>> It is also increasing the chances for flakiness/timeouts.
>> Maybe we could send that feedback.
>>
>> WPT infra could also be improved: more verbose debug-only output,
>> enabling running selected subtest only...
>> testharness.js is actively maintained and is continuously improving.
>> Since we have specific requirements as you are describing, we should push
>> them.
>>
>
> Concretely, please file issues with the "infra" label
> <https://github.com/w3c/web-platform-tests/issues?q=is%3Aissue+is%3Aopen+label%3Ainfra> to
> track upstream WPT infrastructure bugs and feature requests.  Google has an
> ongoing contract with Bocoup (Bob and Mike cc'd) who are improving the
> infrastructure every day.  Of course there's an opportunity to make it even
> better fast with more funding from additional organizations who would
> benefit :-)
>
>>
>> I don't know if there is any way to adopt WPT that won't make testing
>>> less effective. WPT tests may be useful in very rare cases, where we
>>> actively want to communicate certain subtle behavior details to other
>>> vendors - but even so, I imagine that other vendors may not put high
>>> priority on those, for the same reasons.
>>>
>>
>> My own experience is that WPT tests are actually very valuable, at least
>> when we are talking about interoperability/spec conformance.
>> I also see WPT as an efficient way to author tests.
>>
>>
>>>
>>> - Alexey
>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170512/f55624e2/attachment.html>