[webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Dirk Pranke]

On Wed, Apr 11, 2012 at 4:15 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Wed, Apr 11, 2012 at 4:05 PM, Jacob Goldstein <jacobg at adobe.com> wrote:
>>
>> Don't people currently fix tests that are failing and that they didn't
>> author?
>
>
> We don't fix imported tests if you meant modifying html files, etc... to fix
> a broken test file.
>
>> If so, isn't that similar as generating a new ref file in that you first
>> have to understand what is being tested?  There would be the old pixel file
>> to go from, so in many cases this may not be too hard.
>
>
> Yes, but determining whether a pixel result is correct or not is easier than
> determining whether a reference file is correct AND adequately addresses all
> intended test purposes.

It may sometimes be easier to tell if a pixel result is correct or
not, but not always: I've stared at plenty of failing pixel tests
where there were no visible changes at all.

Moreover, I think that requiring a reference file to address "all
intended test purposes" is too high of a bar to allow it to replace a
PNG; I'm not even sure if it's a practical goal at all, especially
given the limited amount of experience we have with reference tests. I
think there is some lower bar of goodness that we can reach in
practice through experience.

-- Dirk