[webkit-dev] Supporting w3c ref tests and changing our convention

Fri Nov 4 12:34:42 PDT 2011

Hi,

There was a discussion about supporting W3C ref tests at TPAC yesterday.
However, there appears to be some disagreement in how we support them.

*Overview*

CSS WG ref tests contain link elements that specify reference files. e.g.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html
<http://december.com/html/4/element/html.html>
xmlns="http://www.w3.org/1999/xhtml">
 <head <http://december.com/html/4/element/head.html>>
  <title <http://december.com/html/4/element/title.html>>CSS Reftest
Reference</title <http://december.com/html/4/element/title.html>>
  <link <http://december.com/html/4/element/link.html> rel="author"
title="NAME_OF_AUTHOR" href="mailto:EMAIL OR http://CONTACT_PAGE"/>
<link <http://december.com/html/4/element/link.html> rel="match"
href="green-box-ref.xht" />
  <style <http://december.com/html/4/element/style.html>
type="text/css"><![CDATA[   CSS FOR REFERENCE  ]]></style
<http://december.com/html/4/element/style.html>>
 </head <http://december.com/html/4/element/head.html>>
 <body <http://december.com/html/4/element/body.html>>
  CONTENT OF REFERENCE
 </body <http://december.com/html/4/element/body.html>></html
<http://december.com/html/4/element/html.html>>

The highlighted line specifies that the rendered result of this page should
be compared with that of green-box-ref.xht. This is very different from our
current method to use filenames such as -match.html, -mismatch.html, etc..
to identify reference files.

In addition, W3C test suites have a build step, which generates
Mozilla-like test manifest files. This file will contain entries that
identifies tests and corresponding reference files. e.g.

== floats-wrap-bfc-outside-001.xht floats-wrap-bfc-outside-001-ref.xht
== floats-wrap-top-below-bfc-001l.xht floats-wrap-top-below-001l-ref.xht
!= floats-wrap-top-below-bfc-001l.xht floats-wrap-top-below-001l-notref.xht
== floats-wrap-top-below-bfc-001r.xht floats-wrap-top-below-001r-ref.xht
!= floats-wrap-top-below-bfc-001r.xht floats-wrap-top-below-001r-notref.xht

All browser vendors who attended the meeting preferred this format.
Microsoft representatives didn't attend the meeting.

*Link element approach*
Pros:

   - Can reuse same ref. file for multiple tests
   - Can have multiple ref. files for single test
   - Information is self-contained in the test file
   - We may get away with test suite build step

(It turns out that we can't convert W3C ref tests to use WebKit conventions
due to the first two points.)

Cons:

   - Requires us modifying each port's DRT to support this format
   - Adding link elements itself may affect tests (all W3C tests are
   required to have link elements at the moment)
   - Hard to understand relationship between files. e.g. if we want to
   figure out which tests use ref.html, we must look at all test files
   - Other browser vendors (Firefox and Opera) prefer manifest file approach

*Manifest file approach*
Pros:

   - Easy to parse, and do not require us modifiying each port's DRT to
   support
   - Easy to tell relationships between files at glance
   - Do not affect tests since manifest files are external to tests
   - Preferred approach of Firefox and Opera

Cons:

   - Have to maintain information in two places, tests and manifests (W3C
   test suites auto-generates manifest files in their build step)
   - Require build step (new W3C tests only have link elements but not
   manifests)

I strongly prefer WebKit use the manifest file approach and deprecate our
current file-name convention because it won't burden each port, and it's
the preferred method of other browser vendors.  Any opinions?
*
*
Best,
Ryosuke Niwa
Software Engineer
Google Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111104/edaf674c/attachment.html>