[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[youenn fablet]

I too am a big proponent of us moving more and more towards WPT.
As part of the streams and fetch API implementation, most of the related
functional tests have been made as WPT.
The benefits of having tests being improved and updated largely outweighs
the testharness.js initial cost.
Somehow, these tests are becoming part of their related specs.

The two complaints I heard against testharness.js are:
- They are less easy to debug as test harness.js does not print out
messages for each assert. There might be room for updating testharness to
support that
- Async tests are less easy to write. While this is probably true,
testharness has support for promise-based tests which are not verbose.
WPT has also some benefits of its own, like the possibility to run easily
the same tests in worker/window environments, the flexible python-based
server...

One complaint I heard against WPT tests is that they require being run
behind a server.
This is becoming more and more true.
There is still a large portion of tests that could be run locally if we
update the paths to test harness.js/testharnessreport,js.
I am not a big fan of modify any WPT test but maybe we should consider
switching back to modifying these paths at import and export time.

I would also like to go in a direction where writing WPT tests is the
default for WebKit layout test.
For that to happen, we need an easy way to upstream tests from WebKit to
WPT GitHub.

I would tend to do the following:
- Write/submit WPT tests in WebKit as regular WebKit testharness.js layout
tests through bugzilla
- At commit queue time, automatically create a PR to WPT GitHub containing
the changes of the WebKit patch
- Ask committer to fix the WPT PR should there be any issue with it
(committer being informed of such issues through bugzilla).

Le dim. 5 févr. 2017 à 15:42, Ryosuke Niwa <rniwa at webkit.org> a écrit :

> Hi all,
>
> *Background*
>
> Web Platform Tests is a suite of tests written for W3C specifications
> <https://github.com/w3c/web-platform-tests/>=,
> and Blink, Edge, Gecko, and WebKit all run them as a part of their
> continuous build system.
>
> Historically, each working group had their own repository for tests,
> but with CSS WG's tests <https://github.com/w3c/csswg-test> finally
> getting merged into the Web Platform Tests,
> we will have a single repository with all the tests from W3C.
>
> A few of us attended a meeting to discuss the future of this test suite
> <https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE/edit#heading=h.aha7njnu40vz>
> last Monday.
> One of the major topic was that Blink and Gecko have adopted the process
> to write the tests
> using testharness.js in their own test suites, and automatically merge
> into Web Platform Tests
> without having to go through another round of code review for Web Platform
> Tests.
>
> Given we do test-driven development in WebKit, I think WebKit should do
> the same.
> This will benefit other browser vendors to catch their bugs with our
> tests, and we will also benefit
> from other browser vendors "reviewing" our tests against their
> implementation and specifications.
>
> In the long term, I believe this will drastically improve the
> interoperability of the Web,
> and dramatically improve the quality of the tests we run against WebKit.
>
> In fact, there was a general consensus that we should even upstream
> pass-if-not-crash tests
> as well as style and layout invalidation tests to web-platform-tests as
> they tend to be undertested
> right now and almost all engines have bugs in this area.
>
> *Problems & Questions*
>
> In order to auto-merge tests from WebKit to Web Platform Tests, there are
> a few problems.
>
> *1. We need to start writing more if not all tests using testharness.js
> instead of our own js-test(-pre).js*
>
> I've heard of many complaints about testharness.js being too verbose and
> cumbersome to use.
> I agree with all those complaints but I don't think we can convince all
> other browser vendors
> to use our own testing script either since both Blink & Gecko are onboard
> with using testharness.js
>
> At this point, I'd argue that the benefit outweighs the cost of adopting
> testharness.js.
> Also, W3C have dropped many styling requirements for tests so we no longer
> need to have
> link/meta elements, etc... You simply include testharness.js and
> testharness-report.js.
>
> See my recent PR
> <https://github.com/w3c/web-platform-tests/commit/c4725e56c91954eaf51759feabec8c6560f10ade> for
> example.
>
> Do people still object to using testharness.js? If so, what are your
> reasons?
>
>
> *2. Where should we put upstremable tests?*
>
> We need a mechanism to identify & merge tests back from WebKit into Web
> Platform Tests.
>
> One way to do this is to start adding tests directly into
> LayoutTests/imported/w3c/web-platform-tests
> and then automatically identify those tests and upstream them.
>
> Assuming all upstream merges happen in timely happen, this should work
> fine.
> And it has a benefit that you get to see all related tests in one place.
>
> Another approach is to create another top-level directory like
> LayoutTests/exports/web-platform-tests.
> One downside of this approach is that we need to match the directory
> structure
> in order for any script to upstream tests to appropriate locations.
>
> Do people prefer one approach over another? Or have another idea?
>
>
> *3. What would be the process for upstreaming tests?*
>
> One possible approach is to create a PR request for every patch posted on
> Bugzilla
> with tests that can be merged into Web Platform Tests.
>
> Because we don't want to make Web Platform Tests less reliably,
> every PR request to Web Platform Tests go through Travis CI
> <https://travis-ci.org/>
> which runs the test hundreds of times to make sure it's not flaky.
>
> This Travis CI job (stability check) will then show up as an additional
> EWS bubble.
>
>
> Another approach is to wait until a patch is landed, and automatically
> create a PR
> for tests that can be upstream'ed. This has a downside that the stability
> check
> wouldn't run until the patch is landed, and it's unclear what happens they
> do fail.
>
> Do committers get emails about them? Do the patches then need to be rolled
> back?
> If we don't come up with an appropriate process, we can end up with a lot
> of
> broken PRs that never get fixed like those failing test expectations :(
>
>
> Again, do you prefer one or the other? Or do you have another proposal?
>
> - R. Niwa
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170206/752aa6a6/attachment.html>