[webkit-dev] Cross browser automated tests

[Maciej Stachowiak]

On May 18, 2008, at 5:09 PM, Sylvain Pasche wrote:

> Hi,
>
> I'm working on an experimental project to build a set of cross browser
> automated tests. The idea would be: having a repository of browser
> independent automated tests. Each browser developer could contribute
> tests to it and use these tests for testing their product. Existing  
> test
> suites that could be run in an automated way could be imported into it
> (something that has already been done by Mozilla/WebKit for the dom
> tests for instance). The technical details of the testing environment
> and API would need to be discussed.
>
> These tests could be run automatically on today's available browsers.
> With the test results published, Web developers could check what  
> feature
> (if covered by tests) are implemented (and to what extent) in a given
> browser.
>
> What do you think about it? Would there be interest in such a project
> from WebKit?

We'd be glad to include any additional tests generated by the project,  
and certainly anyone is welcome to use our tests.

However, we would also want automated regression tests that continue  
to use our harness, which can check many things automatically that are  
not readily testable in-browser, and which is optimized for speed (so  
it can run thousands of tests in a couple of minutes instead of  
hours). These qualities are important to us, and I am not sure a cross- 
browser test harness can achieve them.

> I started already to extract automated tests from Mozilla and WebKit  
> and
> run them on a set of browsers. I first tried to see what tests are not
> WebKit specific. I considered tests which are using
> layoutTestController.dumpAsText(). For these tests, I can use  
> something
> like document.body.nodeValue to get a text serialization of the  
> document
> and compare it against the -expected.txt file.

nodeValue won't cut it, but innerText or textContent may work (depends  
on the test).

>
>
> I ran these tests with a few browsers and published results on
> http://www.browsertests.org/. Note that for WebKit I used the Linux
> GtkLauncher program (some tests made the browser crash). I plan to run
> these tests on Safari Mac/Windows for comparison.
>
> By the way, Mozilla reftest system [1] is interesting in that project
> because it is a browser independent way of testing the layout engine.
> See pages 8-12 on http://www.browsertests.org/test.
>
> I also started a thread on a Mozilla group, visible at [2]
>
>
> Sylvain
>
> [1] http://mxr.mozilla.org/mozilla/source/layout/tools/reftest/README.txt
> [2]
> http://groups.google.com/group/mozilla.dev.quality/browse_frm/thread/b2a959c7547b9877/d5705f4f664bab53
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev