[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[youenn fablet]

The two complaints I heard against testharness.js are:

> - They are less easy to debug as test harness.js does not print out
> messages for each assert. There might be room for updating testharness to
> support that
> - Async tests are less easy to write. While this is probably true,
> testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily
> the same tests in worker/window environments, the flexible python-based
> server...
>
> One complaint I heard against WPT tests is that they require being run
> behind a server.
> This is becoming more and more true.
> There is still a large portion of tests that could be run locally if we
> update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider
> switching back to modifying these paths at import and export time.
>
>
> We should probably do this for the sake of making tests more debuggable.
> Having to run a HTTP/HTTPS server makes testing WebCore against a test
> needlessly harder especially for things like core DOM API which doesn't
> require any network stack.
>

Or we could make our tooling better to also handle LayoutTests/http tests
more easily.
In any case, our CI infrastructure should run WPT tests as closely as
specified by WPT.


> I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
> tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing
> the changes of the WebKit patch
>
>
> I think commit queue time is too late. Remember that not everyone uses
> commit queue. Also, if there was any issue with a test, we should be able
> to tell that prior to landing the patch.
>

Ideally, having one EWS bullet/bot capturing/handling PR status would be
very good.
Knowing the state of a test from various browsers would be helpful at
review time.

But this might require some extra work to support PR updates and so on.
CQ time seems like a reasonable target for step 1.
Well, step 0 might be to have a script that committers would run themselves
locally.


> - Ask committer to fix the WPT PR should there be any issue with it
> (committer being informed of such issues through bugzilla).
>
>
> I think this is too much of an overhead / a burden for WebKit
> contributors. Now patch contributors need to worry about EWS,
> build.webkit.org, and some random PR that gets created on Github failing.
>

Contributors would just need to worry about monitoring bugzilla, like they
are doing for EWS/CQ.
The number of conflicts should be fairly small hopefully.
I am not sure that we can come up with a solution that would handle
conflicts automatically.


> The fact people have to worry about EWS & build.webkit.org separately is
> bad enough. We shouldn't be introducing yet another place people have to
> monitor / care about.
>
> - R. Niwa
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170206/4f602099/attachment.html>