[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Ryosuke Niwa]

On Monday, February 6, 2017, youenn fablet <youennf at gmail.com> wrote:

> The two complaints I heard against testharness.js are:
>
>> - They are less easy to debug as test harness.js does not print out
>>> messages for each assert. There might be room for updating testharness to
>>> support that
>>> - Async tests are less easy to write. While this is probably true,
>>> testharness has support for promise-based tests which are not verbose.
>>> WPT has also some benefits of its own, like the possibility to run
>>> easily the same tests in worker/window environments, the flexible
>>> python-based server...
>>>
>>> One complaint I heard against WPT tests is that they require being run
>>> behind a server.
>>> This is becoming more and more true.
>>> There is still a large portion of tests that could be run locally if we
>>> update the paths to test harness.js/testharnessreport,js.
>>> I am not a big fan of modify any WPT test but maybe we should consider
>>> switching back to modifying these paths at import and export time.
>>>
>>
>> We should probably do this for the sake of making tests more debuggable.
>> Having to run a HTTP/HTTPS server makes testing WebCore against a test
>> needlessly harder especially for things like core DOM API which doesn't
>> require any network stack.
>>
>
> Or we could make our tooling better to also handle LayoutTests/http tests
> more easily.
> In any case, our CI infrastructure should run WPT tests as closely as
> specified by WPT.
>

The concern I've heard about is not how run-webkit-tests run the tests.
It's about how a test is opened inside a browser, DRT, WTR while debugging.


I would tend to do the following:
>>> - Write/submit WPT tests in WebKit as regular WebKit testharness.js
>>> layout tests through bugzilla
>>> - At commit queue time, automatically create a PR to WPT GitHub
>>> containing the changes of the WebKit patch
>>>
>>
>> I think commit queue time is too late. Remember that not everyone uses
>> commit queue. Also, if there was any issue with a test, we should be able
>> to tell that prior to landing the patch.
>>
>
> Ideally, having one EWS bullet/bot capturing/handling PR status would be
> very good.
> Knowing the state of a test from various browsers would be helpful at
> review time.
>
> But this might require some extra work to support PR updates and so on.
> CQ time seems like a reasonable target for step 1.
> Well, step 0 might be to have a script that committers would run
> themselves locally.
>

Again, many people don't use CQ. What do those people do? Manually run a
script? I don't think that's a workable solution. There are many people
just land patches from Xcode UI for example.

In general, I don't think it's acceptable to introduce any extra step for
WebKit contributors. Let it be running a new script, potentially fix an
issue in GitHub PR, etc...

There should be absolutely no new thing contributor has to run or monitor.
That should be the minimum bar for this upstreaming system to exist.

Again, we can't even keep up with test failures on our own bots, and people
frequently land patches with broken tests or tests that fail on some bots.
There is no way we can succeed by introducing yet another step /
place humans has to care. That's simply unacceptable.

- Ask committer to fix the WPT PR should there be any issue with it
>>> (committer being informed of such issues through bugzilla).
>>>
>>
>> I think this is too much of an overhead / a burden for WebKit
>> contributors. Now patch contributors need to worry about EWS,
>> build.webkit.org, and some random PR that gets created on Github failing.
>>
>
> Contributors would just need to worry about monitoring bugzilla, like they
> are doing for EWS/CQ.
>

It depends on what kind of "monitoring" is required.


> The number of conflicts should be fairly small hopefully.
> I am not sure that we can come up with a solution that would handle
> conflicts automatically.
>

In the case of a conflict, the patch should be rejected in EWS & CQ just
the same way a WebKit patch conflict would.

- R. Niwa


-- 
- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170206/ad3752c9/attachment.html>