[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

Sun Feb 5 18:38:24 PST 2017

I also think we should do 2-way sync-ing of WPT tests, similarly to other browser engines. These tests are extremely useful for interoperability. Every major browser engine is involved.

Google also has a nice dashboard showing the pass rates for each WPT test in each major browser. It makes it easier to identify areas we should improve on.

I - for one - would be happy to write tests using testharness.js when it makes sense if it means I can just land them in WebKit with my code change and have them being automagically upstreamed, and run by other browsers as well.

Note that I am not advocating we use testharness.js for everything (although I would be OK with this). I propose we make it opt-in. We have a lot of tests that are not necessarily interesting to upstream. However, the ones that are, using testharness.js makes a lot of sense.

Chris Dumez

> On Feb 5, 2017, at 5:04 PM, youenn fablet <youennf at gmail.com> wrote:
> 
> I too am a big proponent of us moving more and more towards WPT.
> As part of the streams and fetch API implementation, most of the related functional tests have been made as WPT.
> The benefits of having tests being improved and updated largely outweighs the testharness.js initial cost.
> Somehow, these tests are becoming part of their related specs.
> 
> The two complaints I heard against testharness.js are:
> - They are less easy to debug as test harness.js does not print out messages for each assert. There might be room for updating testharness to support that
> - Async tests are less easy to write. While this is probably true, testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily the same tests in worker/window environments, the flexible python-based server...
> 
> One complaint I heard against WPT tests is that they require being run behind a server.
> This is becoming more and more true. 
> There is still a large portion of tests that could be run locally if we update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider switching back to modifying these paths at import and export time.
> 
> I would also like to go in a direction where writing WPT tests is the default for WebKit layout test.
> For that to happen, we need an easy way to upstream tests from WebKit to WPT GitHub.
> 
> I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing the changes of the WebKit patch
> - Ask committer to fix the WPT PR should there be any issue with it (committer being informed of such issues through bugzilla).
> 
>> Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa <rniwa at webkit.org> a écrit :
>> Hi all,
>> 
>> Background
>> 
>> Web Platform Tests is a suite of tests written for W3C specifications=,
>> and Blink, Edge, Gecko, and WebKit all run them as a part of their continuous build system.
>> 
>> Historically, each working group had their own repository for tests,
>> but with CSS WG's tests finally getting merged into the Web Platform Tests,
>> we will have a single repository with all the tests from W3C.
>> 
>> A few of us attended a meeting to discuss the future of this test suite last Monday.
>> One of the major topic was that Blink and Gecko have adopted the process to write the tests
>> using testharness.js in their own test suites, and automatically merge into Web Platform Tests
>> without having to go through another round of code review for Web Platform Tests.
>> 
>> Given we do test-driven development in WebKit, I think WebKit should do the same.
>> This will benefit other browser vendors to catch their bugs with our tests, and we will also benefit
>> from other browser vendors "reviewing" our tests against their implementation and specifications.
>> 
>> In the long term, I believe this will drastically improve the interoperability of the Web,
>> and dramatically improve the quality of the tests we run against WebKit.
>> 
>> In fact, there was a general consensus that we should even upstream pass-if-not-crash tests
>> as well as style and layout invalidation tests to web-platform-tests as they tend to be undertested
>> right now and almost all engines have bugs in this area.
>> 
>> Problems & Questions
>> 
>> In order to auto-merge tests from WebKit to Web Platform Tests, there are a few problems.
>> 
>> 1. We need to start writing more if not all tests using testharness.js instead of our own js-test(-pre).js
>> 
>> I've heard of many complaints about testharness.js being too verbose and cumbersome to use.
>> I agree with all those complaints but I don't think we can convince all other browser vendors
>> to use our own testing script either since both Blink & Gecko are onboard with using testharness.js
>> 
>> At this point, I'd argue that the benefit outweighs the cost of adopting testharness.js.
>> Also, W3C have dropped many styling requirements for tests so we no longer need to have
>> link/meta elements, etc... You simply include testharness.js and testharness-report.js.
>> 
>> See my recent PR for example.
>> 
>> Do people still object to using testharness.js? If so, what are your reasons?
>> 
>> 
>> 2. Where should we put upstremable tests?
>> 
>> We need a mechanism to identify & merge tests back from WebKit into Web Platform Tests.
>> 
>> One way to do this is to start adding tests directly into LayoutTests/imported/w3c/web-platform-tests
>> and then automatically identify those tests and upstream them.
>> 
>> Assuming all upstream merges happen in timely happen, this should work fine.
>> And it has a benefit that you get to see all related tests in one place.
>> 
>> Another approach is to create another top-level directory like LayoutTests/exports/web-platform-tests.
>> One downside of this approach is that we need to match the directory structure
>> in order for any script to upstream tests to appropriate locations.
>> 
>> Do people prefer one approach over another? Or have another idea?
>> 
>> 
>> 3. What would be the process for upstreaming tests?
>> 
>> One possible approach is to create a PR request for every patch posted on Bugzilla
>> with tests that can be merged into Web Platform Tests.
>> 
>> Because we don't want to make Web Platform Tests less reliably,
>> every PR request to Web Platform Tests go through Travis CI
>> which runs the test hundreds of times to make sure it's not flaky.
>> 
>> This Travis CI job (stability check) will then show up as an additional EWS bubble.
>> 
>> 
>> Another approach is to wait until a patch is landed, and automatically create a PR
>> for tests that can be upstream'ed. This has a downside that the stability check
>> wouldn't run until the patch is landed, and it's unclear what happens they do fail.
>> 
>> Do committers get emails about them? Do the patches then need to be rolled back?
>> If we don't come up with an appropriate process, we can end up with a lot of
>> broken PRs that never get fixed like those failing test expectations :(
>> 
>> 
>> Again, do you prefer one or the other? Or do you have another proposal?
>> 
>> - R. Niwa
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170205/33637819/attachment.html>