[webkit-dev] Handling failing reftests

Thu Apr 12 13:46:03 PDT 2012

Right, but when those differences come to light, should the goal be to adjust the ref test to make it platform-independent, or is having platform-specific ref tests acceptable?  Doesn't that put us in the same situation as having platform-specific pixel tests?

From: Ryosuke Niwa <rniwa at webkit.org<mailto:rniwa at webkit.org>>
Date: Thu, 12 Apr 2012 13:43:26 -0700
To: Jacob Goldstein <jacobg at adobe.com<mailto:jacobg at adobe.com>>
Cc: Ojan Vafai <ojan at chromium.org<mailto:ojan at chromium.org>>, Dirk Pranke <dpranke at chromium.org<mailto:dpranke at chromium.org>>, WebKit Development <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>>
Subject: Re: [webkit-dev] Handling failing reftests

Right but you wouldn't know platform-specific issues until you land them. e.g. rounding errors, subtle font differences in edge cases, etc...

On Thu, Apr 12, 2012 at 1:30 PM, Jacob Goldstein <jacobg at adobe.com<mailto:jacobg at adobe.com>> wrote:
Isn't the goal of writing a ref test that it is not platform specific?

From: Ryosuke Niwa <rniwa at webkit.org<mailto:rniwa at webkit.org>>
Date: Thu, 12 Apr 2012 12:29:42 -0700
To: Ojan Vafai <ojan at chromium.org<mailto:ojan at chromium.org>>
Cc: Dirk Pranke <dpranke at chromium.org<mailto:dpranke at chromium.org>>, WebKit Development <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>>
Subject: Re: [webkit-dev] Handling failing reftests

On Thu, Apr 12, 2012 at 11:53 AM, Ojan Vafai <ojan at chromium.org<mailto:ojan at chromium.org>> wrote:
I agree that it's hard sometimes to construct reftests that work, but once you've done so, the cost on the project of maintaining the test is usually considerably lower than pixel tests.

Not so sure. There are cases where we have these platform specific failures for ref tests, and they're much harder to fix than rebaselining pixel results.

- Ryosuke

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120412/002a9af4/attachment.html>