[webkit-dev] naming conventions for reftests and the w3c test suites

[Dirk Pranke]

Hi all,

You may be aware that some people are working on getting w3c-style
reftests to work in our infrastructure (using new-run-webkit-tests).

The few existing reftests we have follow a naming convention of
<testname>-expected.html or <testname>-expected-mismatch.html. This
makes it easy to determine by looking at a directory which files are
tests (vs. expected output or references only), and also which tests
are reftests as opposed to tests that have baselines or reference
output.

The W3C is recommending (at least in the CSS WG) that reference files
that are not themselves tests should be named as <testname>-ref.html
(or the appropriate extension); test files can also live in a
"reftest" subdirectory. (*)

One can debate the various naming conventions; I don't particularly
care what they are as long as they are something consistent, obvious,
and easily automated. However, the naming conventions are currently
not normative; they are a "should" rather than a "must".

I think the "should" should be changed to a "must", and I'd like to
ask this of the testing WG with the WebKit community's endorsement.

Any one object to this or have other thoughts?

-- Dirk

(*) Note that it is acceptable for tests to use other tests as
references, though, so not all reference files will end in -ref.html,
so at least at the moment you can't tell that a file that doesn't end
in -ref.html isn't both a test and a reference. See
http://wiki.csswg.org/test/reftest#the-reftest-reference for more. It
would also be good to allow for a "-notref.html" for expected
mismatches; I'm not sure that that is explicitly standardized, but it
should be. Once we establish a standard, it would also make sense to
rename our existing reftests.