[webkit-dev] Importing W3C tests for editing

[Lukasz Bialek]

Hi all,

 

Some time ago there was a conversation about importing W3C tests into
LayoutTests for WebKit
(https://lists.webkit.org/pipermail/webkit-dev/2013-November/025876.html).
This conversation was paused in November with conclusion, that it may be a
good idea.

 

A brief research showed, that number of editing tests available in
https://github.com/w3c/web-platform-tests is very small. However, there are
a lot of editing tests placed directly in W3C "HTML Editing APIs"
specification
(https://dvcs.w3.org/hg/editing/raw-file/tip/editing.html#conformance-tests)
. It may be a good idea to import those tests instead.

 

There are 90412 tests covering all aspects of W3C specification. Those tests
can be divided into test cases containing several tests with structure
similar to this:

 

PASS [["stylewithcss","true"]] "foo[bar]baz": execCommand("stylewithcss",
false, "true") return value 

PASS [["stylewithcss","true"]] "foo[bar]baz" checks for modifications to
non-editable content 

PASS [["stylewithcss","true"]] "foo[bar]baz" compare innerHTML 

PASS [["stylewithcss","true"]] "foo[bar]baz"
queryCommandIndeterm("stylewithcss") before 

PASS [["stylewithcss","true"]] "foo[bar]baz"
queryCommandState("stylewithcss") before 

FAIL [["stylewithcss","true"]] "foo[bar]baz"
queryCommandValue("stylewithcss") before assert_equals: Wrong result
returned expected "" but got "false"

PASS [["stylewithcss","true"]] "foo[bar]baz"
queryCommandIndeterm("stylewithcss") after 

PASS [["stylewithcss","true"]] "foo[bar]baz"
queryCommandState("stylewithcss") after 

FAIL [["stylewithcss","true"]] "foo[bar]baz"
queryCommandValue("stylewithcss") after assert_equals: Wrong result returned
expected "" but got "true"

 

As a result, we would have 5536 new test files divided into 36 categories:

- backcolor: 70

- bold: 210

- createlink: 48

- defaultparagraphseparator: 11

- delete: 494

- fontname: 141

- fontsize: 156

- forecolor: 147

- formatblock: 362

- forwarddelete: 471

- hilitecolor: 82

- indent: 143

- inserthorizontalrule: 117

- inserthtml: 104

- insertimage: 70

- insertlinebreak: 166

- insertorderedlist: 135

- insertparagraph: 320

- inserttext: 238

- insertunorderedlist: 148

- italic: 143

- justifycenter: 294

- justifyfull: 233

- justifyleft: 149

- justifyright: 233

- outdent: 175

- quasit: 1

- removeformat: 143

- selectall: 1

- strikethrough: 145

- stylewithcss: 17

- subscript: 85

- superscript: 87

- underline: 145

- unlink: 35

- usecss: 17

 

Furthermore, newly created tests has been run and results showed, that
1120/5536 tests run as expected (all sub-tests returned PASS) and 4416/5536
did not. There are 4 WebProcess crushes; 12 tests have some stderr output.
Rest of failures are because of one or more sub-tests failing. Running all
5536 tests takes 32 min 51.978 sec.

 

What do you think about it? Is it worth paying some time for importing those
tests and then aligning WebKit with W3C specification?

 

Best regards,

Lukasz Bialek

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20140205/8236996f/attachment.html>