[webkit-dev] Upstreaming Tests from WebKit to Web Platform Tests

[Ryosuke Niwa]

Hi all,

*Background*

Web Platform Tests is a suite of tests written for W3C specifications
<https://github.com/w3c/web-platform-tests/>=,
and Blink, Edge, Gecko, and WebKit all run them as a part of their
continuous build system.

Historically, each working group had their own repository for tests,
but with CSS WG's tests <https://github.com/w3c/csswg-test> finally getting
merged into the Web Platform Tests,
we will have a single repository with all the tests from W3C.

A few of us attended a meeting to discuss the future of this test suite
<https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE/edit#heading=h.aha7njnu40vz>
last Monday.
One of the major topic was that Blink and Gecko have adopted the process to
write the tests
using testharness.js in their own test suites, and automatically merge into
Web Platform Tests
without having to go through another round of code review for Web Platform
Tests.

Given we do test-driven development in WebKit, I think WebKit should do the
same.
This will benefit other browser vendors to catch their bugs with our tests,
and we will also benefit
from other browser vendors "reviewing" our tests against their
implementation and specifications.

In the long term, I believe this will drastically improve the
interoperability of the Web,
and dramatically improve the quality of the tests we run against WebKit.

In fact, there was a general consensus that we should even upstream
pass-if-not-crash tests
as well as style and layout invalidation tests to web-platform-tests as
they tend to be undertested
right now and almost all engines have bugs in this area.

*Problems & Questions*

In order to auto-merge tests from WebKit to Web Platform Tests, there are a
few problems.

*1. We need to start writing more if not all tests using testharness.js
instead of our own js-test(-pre).js*

I've heard of many complaints about testharness.js being too verbose and
cumbersome to use.
I agree with all those complaints but I don't think we can convince all
other browser vendors
to use our own testing script either since both Blink & Gecko are onboard
with using testharness.js

At this point, I'd argue that the benefit outweighs the cost of adopting
testharness.js.
Also, W3C have dropped many styling requirements for tests so we no longer
need to have
link/meta elements, etc... You simply include testharness.js and
testharness-report.js.

See my recent PR
<https://github.com/w3c/web-platform-tests/commit/c4725e56c91954eaf51759feabec8c6560f10ade>
for
example.

Do people still object to using testharness.js? If so, what are your
reasons?


*2. Where should we put upstremable tests?*

We need a mechanism to identify & merge tests back from WebKit into Web
Platform Tests.

One way to do this is to start adding tests directly into
LayoutTests/imported/w3c/web-platform-tests
and then automatically identify those tests and upstream them.

Assuming all upstream merges happen in timely happen, this should work fine.
And it has a benefit that you get to see all related tests in one place.

Another approach is to create another top-level directory like
LayoutTests/exports/web-platform-tests.
One downside of this approach is that we need to match the directory
structure
in order for any script to upstream tests to appropriate locations.

Do people prefer one approach over another? Or have another idea?


*3. What would be the process for upstreaming tests?*

One possible approach is to create a PR request for every patch posted on
Bugzilla
with tests that can be merged into Web Platform Tests.

Because we don't want to make Web Platform Tests less reliably,
every PR request to Web Platform Tests go through Travis CI
<https://travis-ci.org/>
which runs the test hundreds of times to make sure it's not flaky.

This Travis CI job (stability check) will then show up as an additional EWS
bubble.


Another approach is to wait until a patch is landed, and automatically
create a PR
for tests that can be upstream'ed. This has a downside that the stability
check
wouldn't run until the patch is landed, and it's unclear what happens they
do fail.

Do committers get emails about them? Do the patches then need to be rolled
back?
If we don't come up with an appropriate process, we can end up with a lot of
broken PRs that never get fixed like those failing test expectations :(


Again, do you prefer one or the other? Or do you have another proposal?

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20170205/387286a8/attachment.html>