[webkit-dev] Using W3C test suite as LayoutTests

[youenn fablet]

Hi all,

I made some progress to enable the use of W3C test suite within WebKit.
Some patches are uploaded for that purpose: 222087 233360 233362
233047 235009 235011 235012 235013.
https://github.com/canonresearchfrance/webkit/wiki/Running-W3C-Tests
is updated wth all details.

After some discussions, the idea is to import tests as regular webkit
tests with their expected.txt and test expectations. To automate this
process, two tools are introduced by these patches:
- a first tool (run-w3c-conformance) checks W3C conformance from
generated expected.txt files and generates a report.
- a second tool (generate-w3c-baseline) based on the first one
generates expected.txt files and related TestExpectations fie

The second tool currently puts expected.txt and test expectation files
in LayoutTests platform-specific folders, to cope with port result
discrepancies (console message...).
Import of html files could also be done at runtime so that html files
never appear in WebKit repo. This would allow each port to bump W3C
test suite revision independently.

Some additional points:
- by default, test import uses a W3C test suite revision specified in
a WPTTestsVersion file. The goal would be to always check in tests
from a single revision.
- The testharness.js file used for W3C tests is the one from W3C
repository, The testharnessreport.js file is coming from WebKit
LayoutTests/resources folder.
- The only modification to the test runtime infrastructure is the use
of the WPT server (patch 233360). This is at least needed for HTTP and
WS tests, probably not to most other tests.
- Some tests are flaky due to failed assertions with dynamically
generated values. The tool runs the tests twice to identify and skip
them. These test assertions should probably be fixed.

I plan to start experimenting with this approach by uploading some
tests for the GTK port.
Any feedback or help would be well appreciated.

Regards,
  Youenn

2014-06-19 22:59 GMT+02:00 youenn fablet <youennf at gmail.com>:
> Hi all,
>
> I drafted in
> https://github.com/canonresearchfrance/webkit/wiki/Running-W3C-Tests some
> thoughts about how to run W3C tests within WebKit, the goal being to improve
> regression checks and monitor overall W3C conformance..
>
> It contains how to run the W3C tests using the WPT test server through
> run-webkit-tests (patches available in bugs.webkit.org).
>
> It also describes how to assess results in the absence of any test baseline
> checked in WebKit trunk (the W3C test suite is continuously moving
> anyway...).
>
> Any feedback well appreciated.
>
> Regards,
>    Youenn