[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Ryosuke Niwa]

On Sun, Feb 5, 2017 at 5:04 PM, youenn fablet <youennf at gmail.com> wrote:

> I too am a big proponent of us moving more and more towards WPT.
> As part of the streams and fetch API implementation, most of the related
> functional tests have been made as WPT.
> The benefits of having tests being improved and updated largely outweighs
> the testharness.js initial cost.
> Somehow, these tests are becoming part of their related specs.
>
> The two complaints I heard against testharness.js are:
> - They are less easy to debug as test harness.js does not print out
> messages for each assert. There might be room for updating testharness to
> support that
> - Async tests are less easy to write. While this is probably true,
> testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily
> the same tests in worker/window environments, the flexible python-based
> server...
>
> One complaint I heard against WPT tests is that they require being run
> behind a server.
> This is becoming more and more true.
> There is still a large portion of tests that could be run locally if we
> update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider
> switching back to modifying these paths at import and export time.
>

We should probably do this for the sake of making tests more debuggable.
Having to run a HTTP/HTTPS server makes testing WebCore against a test
needlessly harder especially for things like core DOM API which doesn't
require any network stack.

I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
> tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing
> the changes of the WebKit patch
>

I think commit queue time is too late. Remember that not everyone uses
commit queue. Also, if there was any issue with a test, we should be able
to tell that prior to landing the patch.

- Ask committer to fix the WPT PR should there be any issue with it
> (committer being informed of such issues through bugzilla).
>

I think this is too much of an overhead / a burden for WebKit contributors.
Now patch contributors need to worry about EWS, build.webkit.org, and some
random PR that gets created on Github failing.

The fact people have to worry about EWS & build.webkit.org separately is
bad enough. We shouldn't be introducing yet another place people have to
monitor / care about.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170206/52334eb8/attachment.html>