[webkit-gtk] Layout tests - the breakdown and generating new results

[Zan Dobersek]

This mail will present the layout tests' approximate statistics,
problems and similar for the WebKit GTK+ port as of September 20th,
2009. Ideologically inspired by Adam Barth's mail[0].

Currently (revision 48568), when running run-webkit-tests, 5667 tests
get tested. They more or less all pass, are trivially fixable or need
test results generated (this is actually similar to the overall status,
with some exceptions).

Also currently, we skip 5584 tests. I'd reckon 4000 to 4500 tests need
expected test results generated. Other fail/crash/timeout because of
missing implementations (or bugs) of various elements in DumpRenderTree,
TestNetscapePlugin or WebKit itself. Some may even pass at this moment,
but are not recognized as such and therefor not enabled.

In this mail I'd like to focus mostly on the generation of expected test
results. This would bring 5584 down quickly and effective. There are,
however, some complications to it.

In my opinion, we should start generating expected results and 'compare'
them with expected results on other platforms. With 'compare' I mean to
semantically and syntactically compare them - whether we are generating
same render objects (RenderBlock, RenderLayer, ...), whether there are
no large differences between their positioning (we surely cannot expect
pixel-by-pixel equal rendering, can we?), if text in lines matches, if
css attributes match etc.

This could easily be done with a simple Python script or similar. It
should be fun. Before that, though, I believe we should implement usage
of WebKit test fonts in the DumpRenderTree, similar to what Qt's doing.

Another aspect of testing that we should probably look at is pixel
tests. This could be even more fun than generating expected results.
We'd need to store the WebKitWebView into a pixbuf and save that as a
PNG. There could be some issues when running these tests because of the
themed buttons, scrollbar etc, but I believe it is avoidable with
setting a unique style to the Gtk widget. This one is almost a
guaranteed fun. :)

Overall, there's much to do with tests on the GTK+ port. Though, to
lighten things up a bit, there's already been great improvement with
implementations of accessibility controller, event sender and others.

Regards,
Zan Dobersek.

[0]
https://lists.webkit.org/pipermail/webkit-dev/2009-September/009992.html