[webkit-dev] Another WPT bite

[Ryosuke Niwa]

On Fri, May 12, 2017 at 12:04 PM, Alexey Proskuryakov <ap at webkit.org> wrote:
>
> 12 мая 2017 г., в 11:52, Ben Kelly <ben at wanderview.com> написал(а):
>
> On Fri, May 12, 2017 at 2:26 PM, Rick Byers <rbyers at chromium.org> wrote:
>>
>> On Fri, May 12, 2017 at 2:06 PM, Alexey Proskuryakov <ap at webkit.org>
>> wrote:
>>>
>>> Since imported WPT tests are very flaky, and are not necessarily written
>>> to defend against important regressions, investigating issues with them is
>>> relatively lower priority than investigating issues observed with WebKit
>>> tests. So I would recommend not mixing tests for WebKit regressions with WPT
>>> tests - if your test eventually ends up in LayoutTests/imported, it will
>>> become a lot less effective.
>>
>>
>> FWIW this is absolutely NOT how we're treating this in chromium.  If this
>> is how things end up in practice then we will have failed massively in this
>> effort.
>>
>> We figure if we want the web to behave consistently, we really have no
>> choice but to treat web-platform-tests as first class with all the
>> discipline we give to our own tests.  As such we are actively moving many of
>> our LayoutTests to web-platform-tests and depending entirely on the
>> regression prevention they provide us from there.  Obviously there will be
>> hiccups, but because our product quality will depend on web-platform-tests
>> being an effective and non-flaky testsuite (and because we're starting to
>> require most new features have web-platform-tests before they ship), I'm
>> confident that we've got the incentives in place to lead to constant
>> ratcheting up the engineering discipline and quality of the test suite.
>
>
> FWIW, mozilla also treats WPT as first class tests.  We're not actively
> moving old tests to WPT like google, but all new tests (at least in DOM) are
> being written in WPT.  Of course, we do have exceptions for some tests that
> require gecko-specific features (forcing GC, etc).
>
>
> We don't have a concept of "first class", but I hope that when choosing
> between looking into a simple test that was added for a known important bug,
> and looking into an imported test whose importance is unclear, any WebKit
> engineer will pick the former. And since no one can fix all the things, such
> prioritization makes imported tests less effective.

This is absolutely not how I operate at all. Since almost all custom
elements and shadow DOM API tests I wrote are written using
testharness.js and upstreamed to web-platform-tests, they have been
deleted from LayoutTests/fast/shadow-dom &
LayoutTests/fast/custom-elements.

As such, if any new shadow DOM or custom elements tests under
LayoutTests/imported/ start to fail, then we must fix them since we
idon'thave any other test coverage.

- R. Niwa