[webkit-dev] Cross browser automated tests

[Sylvain Pasche]

Hi,

I'm working on an experimental project to build a set of cross browser 
automated tests. The idea would be: having a repository of browser 
independent automated tests. Each browser developer could contribute 
tests to it and use these tests for testing their product. Existing test 
suites that could be run in an automated way could be imported into it 
(something that has already been done by Mozilla/WebKit for the dom 
tests for instance). The technical details of the testing environment 
and API would need to be discussed.

These tests could be run automatically on today's available browsers. 
With the test results published, Web developers could check what feature 
(if covered by tests) are implemented (and to what extent) in a given 
browser.

What do you think about it? Would there be interest in such a project 
from WebKit?

I started already to extract automated tests from Mozilla and WebKit and 
run them on a set of browsers. I first tried to see what tests are not 
WebKit specific. I considered tests which are using 
layoutTestController.dumpAsText(). For these tests, I can use something 
like document.body.nodeValue to get a text serialization of the document 
and compare it against the -expected.txt file.

I ran these tests with a few browsers and published results on 
http://www.browsertests.org/. Note that for WebKit I used the Linux 
GtkLauncher program (some tests made the browser crash). I plan to run 
these tests on Safari Mac/Windows for comparison.

By the way, Mozilla reftest system [1] is interesting in that project 
because it is a browser independent way of testing the layout engine. 
See pages 8-12 on http://www.browsertests.org/test.

I also started a thread on a Mozilla group, visible at [2]


Sylvain

[1] http://mxr.mozilla.org/mozilla/source/layout/tools/reftest/README.txt
[2] 
http://groups.google.com/group/mozilla.dev.quality/browse_frm/thread/b2a959c7547b9877/d5705f4f664bab53