I thought it would be useful to give a brief overview of the W3C/NIST DOM Test Suites before considering how they could be integrated into WebKit testing. All statements in this message are personal opinions and observations and may not necessarily reflect those of the W3C DOM Interest Group. The W3C/NIST DOM Test Suite (http://www.w3.org/DOM/Test) is an open- source software project of the W3C licensed under the W3C Software License. Distinct test suites exist for each module and level combination. For example, there is a DOM Level 1 Core test suite, a DOM Level 2 HTML test suite and a DOM Level 3 Load and Save Test Suite. With the exception of DOM Level 2 HTML, the test suites do not attempt to duplicate tests in lower levels or other modules. DOM Level 2 HTML is an exception since the recommendation was not upwards compatible with DOM Level 1 HTML and it contains all the tests from the L1 HTML test suite. To claim conformance with the L2 HTML recommendation, an implementation should pass the L2 HTML, L2 Core and L1 Core test suites. Tests are written in XML and conform to a schema that is generated from the XML sources of the DOM recommendations in that level and some simple common language constructs. Download a revision from http://dev.w3.org/cvsweb/2001/DOM-Test-Suite/tests/level1/core/ attrname.xml and view in a text editor. The tests contain metadata describing the source and intention of the tests and link to relevant sections of the recommendation. The tests are written from the perspective of a client of a synchronous parser, that is typically request a parser to load a document and expect the document to be fully loaded on return of control, they perform some action on the returned document and then make assertions. XSLT transforms are used to convert the XML test definitions into Java or ECMAScript (aka Javascript). Other languages have been done outside of the DOM TS effort. The Java tests are typically run using JUnit (http://www.junit.org). The framework was designed to support multiple Java unit test frameworks, however since the initial development the other contenders are no longer under active development. The framework also supports multiple document loaders. The JAXP loaders is the most commonly used, but custom loaders are provided for implementations, such as Batik (SVG renderer), that do not support JAXP. Things get a lot messier with ECMAScript. Parsers may be HTML-only, parsers are usually asynchronous, and there is not a widely deployed parsing API like JAXP. The original NIST Core Test Suite were written as operations on XML documents (staff.xml, staffNS.xml, etc) that contained elements like <employee>, <salary>, etc. To allow the tests to be run against HTML- only implementations, "HTML compatible" variants of the tests (file names start with hc_) were created where XML tag names were replaced with HTML tag names with compatible content models (for example <employee> became <p> and <employeeId> became <em>). The hc_* tests can be run by both XML and HTML processors. The test framework selects an appropriate extension for the test, for example, hc_attrname.xml contains: <load var="doc" href="hc_staff" willBeModified="false"/> The test framework determines whether to load "hc_staff.xml", "hc_staff.html", "hc_staff.xhtml" or "hc_staff.svg". To make the test work with asynchronous parsers, the tests are split into two sections, a request phase that starts parsing a document and a second phase that is started after the document has been completely loaded. The ECMAScript productions come in a two flavors: A JSUnit-hosted (http://www.jsunit.net) test suite and a self-hosted test suites. The JSUnit test suite can be configured to use several different approaches to loading the test documents. The most common for HTML browsers is the IFRAME loader which manipulates documents that are loaded by setting the src attribute on IFRAME elements that are embedded in the JSUnit test definition. Unfortunately, Safari and Konqueror do not dispatch the "load" event to the document containing the IFRAME element which causes the JSUnit test suite to think that the test documents never load. The "self-hosted" productions were created to test implementations that could not run JSUnit, particularly the Palm OS browser (resource issues), ie:Mac Safari/Konqueror (the load event issue), and SVG browsers. The self-hosted tests embed the test script into the test document and the script is run when the document completes loading and then change the document to indicate the success or failure of the test. For example, the SVG tests turn into a green rectangle if pass, red if failed. This approach allows most tests to be run in browsers that don't support JSUnit, however tests that depend on loading multiple documents can't be produced. The DOM TS web page contains the Java and JSUnit productions, however the self-hosted tests are only available by building from the CVS or downloading old and unofficial builds from http://homepage.mac.com/ curt.arnold/FileSharing1.html. Some links: http://lists.w3.org/Archives/Public/www-dom-ts/2005Mar/0002 http://lists.w3.org/Archives/Public/www-dom-ts/2004Mar/0034 So to recap: Tests in XML Converted to Java and ECMAScript JSUnit runs in most browsers, but not Safari or Konqueror (at least last time I checked) Self-hosted tests run in more browsers but doesn't support the few multiple document tests. The self-hosted tests could be a good fit for the existing layout tests. In addition, it may be worthwhile to explore producing either C or Objective-C rendering of the tests if that would make it easier to debug the implementation. I'll attempt to rerun the survey of DOM implementations and provide more specific instructions on building and running the test suites with Safari. The most significant deviation between Java and ECMAScript implementations is that the recommendation is explicit that certain actions should raise exceptions. Java implementation typically do throw the expected exceptions, ECMAScript implementations typically do not, presumably to preserve compatibility with pre-standard implementations and to avoid ever throwing an alert box up to the user. It would be nice if some way of reconciling the compatibility behavior with the recommendation specified behavior. Possibly, that the exception would only be thrown if there is an non-default exception handler. However, that is likely a discussion for much later.