[webkit-dev] W3C/NIST DOM Test Suites: Background Info

[Curt Arnold]

I thought it would be useful to give a brief overview of the W3C/NIST  
DOM Test Suites before considering how they could be integrated into  
WebKit testing.  All statements in this message are personal opinions  
and observations and may not necessarily reflect those of the W3C DOM  
Interest Group.

The W3C/NIST DOM Test Suite (http://www.w3.org/DOM/Test) is an open- 
source software project of the W3C licensed under the W3C Software  
License.  Distinct test suites exist for each module and level  
combination.  For example, there is a DOM Level 1 Core test suite, a  
DOM Level 2 HTML test suite and a DOM Level 3 Load and Save Test  
Suite.  With the exception of DOM Level 2 HTML, the test suites do  
not attempt to duplicate tests in lower levels or other modules.  DOM  
Level 2 HTML is an exception since the recommendation was not upwards  
compatible with DOM Level 1 HTML and it contains all the tests from  
the L1 HTML test suite.  To claim conformance with the L2 HTML  
recommendation, an implementation should pass the L2 HTML, L2 Core  
and L1 Core test suites.

Tests are written in XML and conform to a schema that is generated  
from the XML sources of the DOM recommendations in that level and  
some simple common language constructs.  Download a revision from  
http://dev.w3.org/cvsweb/2001/DOM-Test-Suite/tests/level1/core/ 
attrname.xml and view in a text editor.  The tests contain metadata  
describing the source and intention of the tests and link to relevant  
sections of the recommendation.  The tests are written from the  
perspective of a client of a synchronous parser, that is typically  
request a parser to load a document and expect the document to be  
fully loaded on return of control, they perform some action on the  
returned document and then make assertions.

XSLT transforms are used to convert the XML test definitions into  
Java or ECMAScript (aka Javascript).  Other languages have been done  
outside of the DOM TS effort.  The Java tests are typically run using  
JUnit (http://www.junit.org).  The framework was designed to support  
multiple Java unit test frameworks, however since the initial  
development the other contenders are no longer under active  
development.  The framework also supports multiple document loaders.   
The JAXP loaders is the most commonly used, but custom loaders are  
provided for implementations, such as Batik (SVG renderer), that do  
not support JAXP.

Things get a lot messier with ECMAScript.  Parsers may be HTML-only,  
parsers are usually asynchronous, and there is not a widely deployed  
parsing API like JAXP.

The original NIST Core Test Suite were written as operations on XML  
documents (staff.xml, staffNS.xml, etc) that contained elements like  
<employee>, <salary>, etc.  To allow the tests to be run against HTML- 
only implementations, "HTML compatible" variants of the tests (file  
names start with hc_) were created where XML tag names were replaced  
with HTML tag names with compatible content models (for example  
<employee> became <p> and <employeeId> became <em>).   The hc_* tests  
can be run by both XML and HTML processors.  The test framework  
selects an appropriate extension for the test, for example,  
hc_attrname.xml contains:

<load var="doc" href="hc_staff" willBeModified="false"/>

The test framework determines whether to load "hc_staff.xml",  
"hc_staff.html", "hc_staff.xhtml" or "hc_staff.svg".

To make the test work with asynchronous parsers, the tests are split  
into two sections, a request phase that starts parsing a document and  
a second phase that is started after the document has been completely  
loaded.

The ECMAScript productions come in a two flavors: A JSUnit-hosted  
(http://www.jsunit.net) test suite and a self-hosted test suites.   
The JSUnit test suite can be configured to use several different  
approaches to loading the test documents.  The most common for HTML  
browsers is the IFRAME loader which manipulates documents that are  
loaded by setting the src attribute on IFRAME elements that are  
embedded in the JSUnit test definition.  Unfortunately, Safari and  
Konqueror do not dispatch the "load" event to the document containing  
the IFRAME element which causes the JSUnit test suite to think that  
the test documents never load.

The "self-hosted" productions were created to test implementations  
that could not run JSUnit, particularly the Palm OS browser (resource  
issues), ie:Mac Safari/Konqueror (the load event issue), and SVG  
browsers.  The self-hosted tests embed the test script into the test  
document and the script is run when the document completes loading  
and then change the document to indicate the success or failure of  
the test.  For example, the SVG tests turn into a green rectangle if  
pass, red if failed.  This approach allows most tests to be run in  
browsers that don't support JSUnit, however tests that depend on  
loading multiple documents can't be produced.

The DOM TS web page contains the Java and JSUnit productions, however  
the self-hosted tests are only available by building from the CVS or  
downloading old and unofficial builds from http://homepage.mac.com/ 
curt.arnold/FileSharing1.html.

Some links:

http://lists.w3.org/Archives/Public/www-dom-ts/2005Mar/0002
http://lists.w3.org/Archives/Public/www-dom-ts/2004Mar/0034


So to recap:

Tests in XML
Converted to Java and ECMAScript
JSUnit runs in most browsers, but not Safari or Konqueror (at least  
last time I checked)
Self-hosted tests run in more browsers but doesn't support the few  
multiple document tests.

The self-hosted tests could be a good fit for the existing layout  
tests.  In addition, it may be worthwhile to explore producing either  
C or Objective-C rendering of the tests if that would make it easier  
to debug the implementation.

I'll attempt to rerun the survey of DOM implementations and provide  
more specific instructions on building and running the test suites  
with Safari.

The most significant deviation between Java and ECMAScript  
implementations is that the recommendation is explicit that certain  
actions should raise exceptions.  Java implementation typically do  
throw the expected exceptions, ECMAScript implementations typically  
do not, presumably to preserve compatibility with pre-standard  
implementations and to avoid ever throwing an alert box up to the  
user.  It would be nice if some way of reconciling the compatibility  
behavior with the recommendation specified behavior.  Possibly, that  
the exception would only be thrown if there is an non-default  
exception handler.  However, that is likely a discussion for much later.