[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

Ryosuke Niwa rniwa at webkit.org
Sun Feb 5 15:41:19 PST 2017

Hi all,


Web Platform Tests is a suite of tests written for W3C specifications
and Blink, Edge, Gecko, and WebKit all run them as a part of their
continuous build system.

Historically, each working group had their own repository for tests,
but with CSS WG's tests <https://github.com/w3c/csswg-test> finally getting
merged into the Web Platform Tests,
we will have a single repository with all the tests from W3C.

A few of us attended a meeting to discuss the future of this test suite
last Monday.
One of the major topic was that Blink and Gecko have adopted the process to
write the tests
using testharness.js in their own test suites, and automatically merge into
Web Platform Tests
without having to go through another round of code review for Web Platform

Given we do test-driven development in WebKit, I think WebKit should do the
This will benefit other browser vendors to catch their bugs with our tests,
and we will also benefit
from other browser vendors "reviewing" our tests against their
implementation and specifications.

In the long term, I believe this will drastically improve the
interoperability of the Web,
and dramatically improve the quality of the tests we run against WebKit.

In fact, there was a general consensus that we should even upstream
pass-if-not-crash tests
as well as style and layout invalidation tests to web-platform-tests as
they tend to be undertested
right now and almost all engines have bugs in this area.

*Problems & Questions*

In order to auto-merge tests from WebKit to Web Platform Tests, there are a
few problems.

*1. We need to start writing more if not all tests using testharness.js
instead of our own js-test(-pre).js*

I've heard of many complaints about testharness.js being too verbose and
cumbersome to use.
I agree with all those complaints but I don't think we can convince all
other browser vendors
to use our own testing script either since both Blink & Gecko are onboard
with using testharness.js

At this point, I'd argue that the benefit outweighs the cost of adopting
Also, W3C have dropped many styling requirements for tests so we no longer
need to have
link/meta elements, etc... You simply include testharness.js and

See my recent PR

Do people still object to using testharness.js? If so, what are your

*2. Where should we put upstremable tests?*

We need a mechanism to identify & merge tests back from WebKit into Web
Platform Tests.

One way to do this is to start adding tests directly into
and then automatically identify those tests and upstream them.

Assuming all upstream merges happen in timely happen, this should work fine.
And it has a benefit that you get to see all related tests in one place.

Another approach is to create another top-level directory like
One downside of this approach is that we need to match the directory
in order for any script to upstream tests to appropriate locations.

Do people prefer one approach over another? Or have another idea?

*3. What would be the process for upstreaming tests?*

One possible approach is to create a PR request for every patch posted on
with tests that can be merged into Web Platform Tests.

Because we don't want to make Web Platform Tests less reliably,
every PR request to Web Platform Tests go through Travis CI
which runs the test hundreds of times to make sure it's not flaky.

This Travis CI job (stability check) will then show up as an additional EWS

Another approach is to wait until a patch is landed, and automatically
create a PR
for tests that can be upstream'ed. This has a downside that the stability
wouldn't run until the patch is landed, and it's unclear what happens they
do fail.

Do committers get emails about them? Do the patches then need to be rolled
If we don't come up with an appropriate process, we can end up with a lot of
broken PRs that never get fixed like those failing test expectations :(

Again, do you prefer one or the other? Or do you have another proposal?

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170205/387286a8/attachment.html>

More information about the webkit-dev mailing list