[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

Sun Feb 5 18:54:02 PST 2017

FWIW, in Blink we stopped rewriting the testharness.js paths before
switching to wptserve, by instead rewriting those URLs only when running
LayoutTests:
https://cs.chromium.org/chromium/src/content/shell/renderer/layout_test/blink_test_runner.cc?type=cs&q=content/shell/renderer/layout_test/blink_test_runner.cc&l=221

So, we know that it's possible to run a lot of the tests without wptserve
without modifying the tests. However, trying to list all the tests that do
or don't need wptserve seems like a lot of work, and I think we're now
using wptserve for all tests.

On Sun, Feb 5, 2017 at 8:05 PM youenn fablet <youennf at gmail.com> wrote:

> I too am a big proponent of us moving more and more towards WPT.
> As part of the streams and fetch API implementation, most of the related
> functional tests have been made as WPT.
> The benefits of having tests being improved and updated largely outweighs
> the testharness.js initial cost.
> Somehow, these tests are becoming part of their related specs.
>
> The two complaints I heard against testharness.js are:
> - They are less easy to debug as test harness.js does not print out
> messages for each assert. There might be room for updating testharness to
> support that
> - Async tests are less easy to write. While this is probably true,
> testharness has support for promise-based tests which are not verbose.
> WPT has also some benefits of its own, like the possibility to run easily
> the same tests in worker/window environments, the flexible python-based
> server...
>
> One complaint I heard against WPT tests is that they require being run
> behind a server.
> This is becoming more and more true.
> There is still a large portion of tests that could be run locally if we
> update the paths to test harness.js/testharnessreport,js.
> I am not a big fan of modify any WPT test but maybe we should consider
> switching back to modifying these paths at import and export time.
>
> I would also like to go in a direction where writing WPT tests is the
> default for WebKit layout test.
> For that to happen, we need an easy way to upstream tests from WebKit to
> WPT GitHub.
>
> I would tend to do the following:
> - Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
> tests through bugzilla
> - At commit queue time, automatically create a PR to WPT GitHub containing
> the changes of the WebKit patch
> - Ask committer to fix the WPT PR should there be any issue with it
> (committer being informed of such issues through bugzilla).
>
> Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa <rniwa at webkit.org> a écrit :
>
> Hi all,
>
> *Background*
>
> Web Platform Tests is a suite of tests written for W3C specifications
> <https://github.com/w3c/web-platform-tests/>=,
> and Blink, Edge, Gecko, and WebKit all run them as a part of their
> continuous build system.
>
> Historically, each working group had their own repository for tests,
> but with CSS WG's tests <https://github.com/w3c/csswg-test> finally
> getting merged into the Web Platform Tests,
> we will have a single repository with all the tests from W3C.
>
> A few of us attended a meeting to discuss the future of this test suite
> <https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE/edit#heading=h.aha7njnu40vz>
> last Monday.
> One of the major topic was that Blink and Gecko have adopted the process
> to write the tests
> using testharness.js in their own test suites, and automatically merge
> into Web Platform Tests
> without having to go through another round of code review for Web Platform
> Tests.
>
> Given we do test-driven development in WebKit, I think WebKit should do
> the same.
> This will benefit other browser vendors to catch their bugs with our
> tests, and we will also benefit
> from other browser vendors "reviewing" our tests against their
> implementation and specifications.
>
> In the long term, I believe this will drastically improve the
> interoperability of the Web,
> and dramatically improve the quality of the tests we run against WebKit.
>
> In fact, there was a general consensus that we should even upstream
> pass-if-not-crash tests
> as well as style and layout invalidation tests to web-platform-tests as
> they tend to be undertested
> right now and almost all engines have bugs in this area.
>
> *Problems & Questions*
>
> In order to auto-merge tests from WebKit to Web Platform Tests, there are
> a few problems.
>
> *1. We need to start writing more if not all tests using testharness.js
> instead of our own js-test(-pre).js*
>
> I've heard of many complaints about testharness.js being too verbose and
> cumbersome to use.
> I agree with all those complaints but I don't think we can convince all
> other browser vendors
> to use our own testing script either since both Blink & Gecko are onboard
> with using testharness.js
>
> At this point, I'd argue that the benefit outweighs the cost of adopting
> testharness.js.
> Also, W3C have dropped many styling requirements for tests so we no longer
> need to have
> link/meta elements, etc... You simply include testharness.js and
> testharness-report.js.
>
> See my recent PR
> <https://github.com/w3c/web-platform-tests/commit/c4725e56c91954eaf51759feabec8c6560f10ade> for
> example.
>
> Do people still object to using testharness.js? If so, what are your
> reasons?
>
>
> *2. Where should we put upstremable tests?*
>
> We need a mechanism to identify & merge tests back from WebKit into Web
> Platform Tests.
>
> One way to do this is to start adding tests directly into
> LayoutTests/imported/w3c/web-platform-tests
> and then automatically identify those tests and upstream them.
>
> Assuming all upstream merges happen in timely happen, this should work
> fine.
> And it has a benefit that you get to see all related tests in one place.
>
> Another approach is to create another top-level directory like
> LayoutTests/exports/web-platform-tests.
> One downside of this approach is that we need to match the directory
> structure
> in order for any script to upstream tests to appropriate locations.
>
> Do people prefer one approach over another? Or have another idea?
>
>
> *3. What would be the process for upstreaming tests?*
>
> One possible approach is to create a PR request for every patch posted on
> Bugzilla
> with tests that can be merged into Web Platform Tests.
>
> Because we don't want to make Web Platform Tests less reliably,
> every PR request to Web Platform Tests go through Travis CI
> <https://travis-ci.org/>
> which runs the test hundreds of times to make sure it's not flaky.
>
> This Travis CI job (stability check) will then show up as an additional
> EWS bubble.
>
>
> Another approach is to wait until a patch is landed, and automatically
> create a PR
> for tests that can be upstream'ed. This has a downside that the stability
> check
> wouldn't run until the patch is landed, and it's unclear what happens they
> do fail.
>
> Do committers get emails about them? Do the patches then need to be rolled
> back?
> If we don't come up with an appropriate process, we can end up with a lot
> of
> broken PRs that never get fixed like those failing test expectations :(
>
>
> Again, do you prefer one or the other? Or do you have another proposal?
>
> - R. Niwa
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170206/f56a4a0e/attachment.html>