[webkit-dev] Handling failing reftests

[Dirk Pranke]

On Thu, Apr 12, 2012 at 12:36 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Thu, Apr 12, 2012 at 12:12 AM, Osztrogonac Csaba <oszi at inf.u-szeged.hu>
> wrote:
>>
>> I ran into a failing reftest and I didn't find a proper way for handling
>> it.
>> fast/multicol/cell-shrinkback.html is a new reftest introduced in
>> http://trac.webkit.org/changeset/113738 .
>> But unfortunately it fails almost everywhere (Chromium, GTK, Qt).
>
>
> What is the failure? Is there a bug?
>
>> Chromium guys added platform specific png files into
>> platform/chromium-linux and platform/chromium-win
>> directories - http://trac.webkit.org/changeset/113790
>
>
> That won't work.
>
>> I tried to do same thing for Qt port, but the test still fails -
>> http://trac.webkit.org/changeset/113869
>> It seems NRWT ignores expected png and txt files for reftests at all.
>
>
> Right because it's a ref test.
>
>> And now I found that it is added to the test_expectations.txt for
>> Chromium:
>>
>> http://trac.webkit.org/changeset/113797/trunk/LayoutTests/platform/chromium/test_expectations.txt
>>
>> So what is the proper way for handling failing reftests? In this case
>> isn't possible
>> if the reftest is incorrect, because it fails almost on all platform, but
>> Mac?
>
>
> You can either skip the test or add a platform-specific reference file (e.g.
> platform/qt/fast/multicol/cell-shrinkback-expected.html).
>

This is an interesting situation; does it make sense to require our
reftests to be generic, or is that unrealistic? I also wonder if we
should allow  platform-specific pixel tests to coexist with
platform-specific reference tests (since you could obviously write a
platform-specific ref test that just displayed a single IMG, but the
tools don't make that particularly easy).

-- Dirk