[webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Dirk Pranke]

On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan <lists at roberthogan.net> wrote:
> On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote:
>>
>> What does "clean up the existing folder" entail?
>>
>
> Just move any -expected.html file out of there and generate pixel results.
> Or move the test and its -expected.html into a folder in fast/css.

Okay, following a conversation on #irc, I think I understand this
better. There do appear to be differences of opinion over what to do
here, so I will try to describe this from ground zero.

The CSS test suite has ~9000 tests. the correctness of which is
established manually (and visually). We would like to import these
tests and run them automatically. Some small number of tests (a few
hundred?) do have reference html, but most don't. Since most tests do
not come with reference html or expected PNGs, we have to add one or
the other.

Currently we have added ~400 or so to the tree, and when we add them
we have been writing -expected.html reference results to avoid
generating pixel results where possible.

This raises at least two questions, which have been confusing the
discussion until now:

1) What is the best way to add these tests to the tree? Should we add
them one-at-a-time with -expected files that we have created? Should
we add them all at once and SKIP the tests until we have generated
-expected results for each test? Or should we only import subsets that
have official -expected files?

2) Is it acceptable for someone other than the test author to write
-expected html references? What bar do we need to have to ensure that
the reference file will catch the bug the initial test is intended to
catch? Do we (or should we) maintain a similar bar for pixel tests
today?

Personally, I don't care that much about (1), as long as we can track
the provenance of the tests (where they came from) ... I would be
happy with us importing tests one at a time and adding our own
results, and hopefully feeding them back up stream to the W3C.

I care much more about (2): I believe we should not be introducing new
tests that only have pixel tests for results unless absolutely
necessary - the cost of maintaining pixel test results vastly
outweighs the potential benefit we get by having a reference file that
won't break the same way the test breaks. I believe it is reasonably
for someone other than the test author to write a reference html for a
test and get it to be good enough that it will provide more value than
a pixel test result, and we should be encouraging that.

I believe we're hoping to discuss this more broadly at the
contributors meeting, but it might be good to discuss here first.
Thoughts?

-- Dirk