[webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Dirk Pranke]

On Wed, Apr 11, 2012 at 4:00 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein <jacobg at adobe.com> wrote:
>>
>> +1 on not introducing new pixel tests and allowing someone other than the
>> test author to create the -expected file.
>>
>> We may also be able to streamline some of this process by implementing
>> some helper scripts.  Ultimately, someone will still have to review new
>> files manually, but scripts should be able to speed up the process.
>
>
> -1 on that. As I said on other threads about this topic, determining whether
> a reference file adequately detect all bugs a test is intended to test is
> hard, and losing the test coverage at the cost of lowering maintenance cost
> is not necessary a good thing.
>
> Also, adding a reference file would mean that either we're adding -ref.html
> / -noref.html files or modifying reftest.list. If doing the former, then we
> can't use this approach in any directory where we use reftest.list at the
> moment because we explicitly prohibit mixing naming convention and
> reftest.list.
>
> Modifying reftest.list is essentially modifying the test suite, and it seems
> like there is a consensus that we don't want to do it.
>

This is a quibble compared to the first paragraph, but the last two
paragraphs are merely implementation details (especially if we think
we're generating reftest.list from <link> tags embedded in the tests).
I think if we agree that it's okay to add -ref.html / -noref.html
files for tests we can revisit what the best way to manage such a
process is. I think the initial guideline was established when we
thought that an imported test suite would come with all of the needed
reference files, and in such a case I agree we should leave it as
"stock" as possible.

-- Dirk