[webkit-dev] naming conventions for reftests and the w3c test suites

[Ryosuke Niwa]

Do we know if Mozilla's test suite follow such a convention? Given we
already have tables/mozilla,
there appears be an interest to import some Mozilla tests to WebKit.
e.g. I'm planning to import Mozilla's
reftests for unicode-bidi: isolate / plaintext as well.

If their test suite don't follow such a convention, I'm not certain if
there's a much benefit in asking this.

Furthermore, if we expect there will be only few reference files that don't
start with ref-, notref-,
don't ends with -ref.*, -notref.* and are not in a subdirectory of a
directory named either "reference" and "reftest",
then we can treat all those files as traditional pixel tests.

- Ryosuke

On Thu, Dec 1, 2011 at 1:58 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> Hi all,
>
> You may be aware that some people are working on getting w3c-style
> reftests to work in our infrastructure (using new-run-webkit-tests).
>
> The few existing reftests we have follow a naming convention of
> <testname>-expected.html or <testname>-expected-mismatch.html. This
> makes it easy to determine by looking at a directory which files are
> tests (vs. expected output or references only), and also which tests
> are reftests as opposed to tests that have baselines or reference
> output.
>
> The W3C is recommending (at least in the CSS WG) that reference files
> that are not themselves tests should be named as <testname>-ref.html
> (or the appropriate extension); test files can also live in a
> "reftest" subdirectory. (*)
>
> One can debate the various naming conventions; I don't particularly
> care what they are as long as they are something consistent, obvious,
> and easily automated. However, the naming conventions are currently
> not normative; they are a "should" rather than a "must".
>
> I think the "should" should be changed to a "must", and I'd like to
> ask this of the testing WG with the WebKit community's endorsement.
>
> Any one object to this or have other thoughts?
>
> -- Dirk
>
> (*) Note that it is acceptable for tests to use other tests as
> references, though, so not all reference files will end in -ref.html,
> so at least at the moment you can't tell that a file that doesn't end
> in -ref.html isn't both a test and a reference. See
> http://wiki.csswg.org/test/reftest#the-reftest-reference for more. It
> would also be good to allow for a "-notref.html" for expected
> mismatches; I'm not sure that that is explicitly standardized, but it
> should be. Once we establish a standard, it would also make sense to
> rename our existing reftests.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111201/4d0466e6/attachment.html>