[webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Jacob Goldstein]

+1 on not introducing new pixel tests and allowing someone other than the
test author to create the -expected file.

We may also be able to streamline some of this process by implementing
some helper scripts.  Ultimately, someone will still have to review new
files manually, but scripts should be able to speed up the process.

Alan Stearns and I plan to present some ideas around W3C and WebKit test
integration at the contributors meeting to elicit more feedback and
recruit people to support the effort.




On 4/11/12 3:29 PM, "Dirk Pranke" <dpranke at chromium.org> wrote:

>On Wed, Apr 11, 2012 at 12:31 PM, Robert Hogan <lists at roberthogan.net>
>wrote:
>> On Wednesday 11 April 2012 20:27:23 Dirk Pranke wrote:
>>>
>>> What does "clean up the existing folder" entail?
>>>
>>
>> Just move any -expected.html file out of there and generate pixel
>>results.
>> Or move the test and its -expected.html into a folder in fast/css.
>
>Okay, following a conversation on #irc, I think I understand this
>better. There do appear to be differences of opinion over what to do
>here, so I will try to describe this from ground zero.
>
>The CSS test suite has ~9000 tests. the correctness of which is
>established manually (and visually). We would like to import these
>tests and run them automatically. Some small number of tests (a few
>hundred?) do have reference html, but most don't. Since most tests do
>not come with reference html or expected PNGs, we have to add one or
>the other.
>
>Currently we have added ~400 or so to the tree, and when we add them
>we have been writing -expected.html reference results to avoid
>generating pixel results where possible.
>
>This raises at least two questions, which have been confusing the
>discussion until now:
>
>1) What is the best way to add these tests to the tree? Should we add
>them one-at-a-time with -expected files that we have created? Should
>we add them all at once and SKIP the tests until we have generated
>-expected results for each test? Or should we only import subsets that
>have official -expected files?
>
>2) Is it acceptable for someone other than the test author to write
>-expected html references? What bar do we need to have to ensure that
>the reference file will catch the bug the initial test is intended to
>catch? Do we (or should we) maintain a similar bar for pixel tests
>today?
>
>Personally, I don't care that much about (1), as long as we can track
>the provenance of the tests (where they came from) ... I would be
>happy with us importing tests one at a time and adding our own
>results, and hopefully feeding them back up stream to the W3C.
>
>I care much more about (2): I believe we should not be introducing new
>tests that only have pixel tests for results unless absolutely
>necessary - the cost of maintaining pixel test results vastly
>outweighs the potential benefit we get by having a reference file that
>won't break the same way the test breaks. I believe it is reasonably
>for someone other than the test author to write a reference html for a
>test and get it to be good enough that it will provide more value than
>a pixel test result, and we should be encouraging that.
>
>I believe we're hoping to discuss this more broadly at the
>contributors meeting, but it might be good to discuss here first.
>Thoughts?
>
>-- Dirk
>_______________________________________________
>webkit-dev mailing list
>webkit-dev at lists.webkit.org
>http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev