[webkit-dev] Handling failing reftests

[Dirk Pranke]

On Apr 12, 2012 2:25 PM, "Ojan Vafai" <ojan at chromium.org> wrote:
>
> On Thu, Apr 12, 2012 at 2:18 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> On Apr 12, 2012 1:53 PM, "Ryosuke Niwa" <rniwa at webkit.org> wrote:
>> >
>> > My suggestion is to try to make it platform-independent but it's
significantly harder than rebaselining the results for pixel tests. As you
point out, it defeats one of the benefits of reftests if we end up
introducing platform-specific reference files.
>> >
>> > - Ryosuke
>> >
>>
>> It defeats one of the benefits but not all, I think (hope?)
>
> Maintaining a reftest with multiple platform-specific baselines seems
like it would be strictly more work than maintaining a pixel test with
multiple platform-specific baselines in most cases.

It's hard to say without specific examples, but I can imagine cases where
if Chromium switched to reftests it would eliminate a large number of
rebaselines caused by Skis changes and it might still be a net win.

That said, I have no doubt that we should try hard to make tests generic
where possible.

-- Dirk
>>
>> -- Dirk
>>
>>
>> >
>> > On Thu, Apr 12, 2012 at 1:46 PM, Jacob Goldstein <jacobg at adobe.com>
wrote:
>> >>
>> >> Right, but when those differences come to light, should the goal be
to adjust the ref test to make it platform-independent, or is having
platform-specific ref tests acceptable?  Doesn't that put us in the same
situation as having platform-specific pixel tests?
>> >>
>> >> From: Ryosuke Niwa <rniwa at webkit.org>
>> >> Date: Thu, 12 Apr 2012 13:43:26 -0700
>> >> To: Jacob Goldstein <jacobg at adobe.com>
>> >> Cc: Ojan Vafai <ojan at chromium.org>, Dirk Pranke <dpranke at chromium.org>,
WebKit Development <webkit-dev at lists.webkit.org>
>> >>
>> >> Subject: Re: [webkit-dev] Handling failing reftests
>> >>
>> >> Right but you wouldn't know platform-specific issues until you land
them. e.g. rounding errors, subtle font differences in edge cases, etc...
>> >>
>> >> On Thu, Apr 12, 2012 at 1:30 PM, Jacob Goldstein <jacobg at adobe.com>
wrote:
>> >>>
>> >>> Isn't the goal of writing a ref test that it is not platform
specific?
>> >>>
>> >>> From: Ryosuke Niwa <rniwa at webkit.org>
>> >>> Date: Thu, 12 Apr 2012 12:29:42 -0700
>> >>> To: Ojan Vafai <ojan at chromium.org>
>> >>> Cc: Dirk Pranke <dpranke at chromium.org>, WebKit Development <
webkit-dev at lists.webkit.org>
>> >>> Subject: Re: [webkit-dev] Handling failing reftests
>> >>>
>> >>> On Thu, Apr 12, 2012 at 11:53 AM, Ojan Vafai <ojan at chromium.org>
wrote:
>> >>>>
>> >>>> I agree that it's hard sometimes to construct reftests that work,
but once you've done so, the cost on the project of maintaining the test is
usually considerably lower than pixel tests.
>> >>>
>> >>>
>> >>> Not so sure. There are cases where we have these platform specific
failures for ref tests, and they're much harder to fix than rebaselining
pixel results.
>> >>>
>> >>> - Ryosuke
>> >>>
>> >>
>> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120412/00680a78/attachment.html>