[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[youenn fablet]

It seems we agree in moving this forward. Any objection?
If so, let's start with small practical steps, something like a script to
automatically generate WPT PR requests from a WebKit repo.

Le lun. 6 févr. 2017 à 12:28, Ryosuke Niwa <rniwa at webkit.org> a écrit :

>
> On Monday, February 6, 2017, youenn fablet <youennf at gmail.com> wrote:
>
> The two complaints I heard against testharness.js are:
>
> - They are less easy to debug as test harness.js does not print out
> messages for each assert. There might be room for updating testharness to
> support that
> - Async tests are less easy to write. While this is probably true,
> testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily
> the same tests in worker/window environments, the flexible python-based
> server...
>
> One complaint I heard against WPT tests is that they require being run
> behind a server.
> This is becoming more and more true.
> There is still a large portion of tests that could be run locally if we
> update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider
> switching back to modifying these paths at import and export time.
>
>
> We should probably do this for the sake of making tests more debuggable.
> Having to run a HTTP/HTTPS server makes testing WebCore against a test
> needlessly harder especially for things like core DOM API which doesn't
> require any network stack.
>
>
> Or we could make our tooling better to also handle LayoutTests/http tests
> more easily.
> In any case, our CI infrastructure should run WPT tests as closely as
> specified by WPT.
>
>
> The concern I've heard about is not how run-webkit-tests run the tests.
> It's about how a test is opened inside a browser, DRT, WTR while debugging.
>
>
> I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
> tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing
> the changes of the WebKit patch
>
>
> I think commit queue time is too late. Remember that not everyone uses
> commit queue. Also, if there was any issue with a test, we should be able
> to tell that prior to landing the patch.
>
>
> Ideally, having one EWS bullet/bot capturing/handling PR status would be
> very good.
> Knowing the state of a test from various browsers would be helpful at
> review time.
>
> But this might require some extra work to support PR updates and so on.
> CQ time seems like a reasonable target for step 1.
> Well, step 0 might be to have a script that committers would run
> themselves locally.
>
>
> Again, many people don't use CQ. What do those people do? Manually run a
> script? I don't think that's a workable solution. There are many people
> just land patches from Xcode UI for example.
>
> In general, I don't think it's acceptable to introduce any extra step for
> WebKit contributors. Let it be running a new script, potentially fix an
> issue in GitHub PR, etc...
>
> There should be absolutely no new thing contributor has to run or monitor.
> That should be the minimum bar for this upstreaming system to exist.
>
> Again, we can't even keep up with test failures on our own bots, and
> people frequently land patches with broken tests or tests that fail on some
> bots. There is no way we can succeed by introducing yet another step /
> place humans has to care. That's simply unacceptable.
>
> - Ask committer to fix the WPT PR should there be any issue with it
> (committer being informed of such issues through bugzilla).
>
>
> I think this is too much of an overhead / a burden for WebKit
> contributors. Now patch contributors need to worry about EWS,
> build.webkit.org, and some random PR that gets created on Github failing.
>
>
> Contributors would just need to worry about monitoring bugzilla, like they
> are doing for EWS/CQ.
>
>
> It depends on what kind of "monitoring" is required.
>
>
> The number of conflicts should be fairly small hopefully.
> I am not sure that we can come up with a solution that would handle
> conflicts automatically.
>
>
> In the case of a conflict, the patch should be rejected in EWS & CQ just
> the same way a WebKit patch conflict would.
>
> - R. Niwa
>
>
> --
> - R. Niwa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170207/84b85693/attachment.html>