[webkit-dev] Handling failing reftests

[Ojan Vafai]

On Thu, Apr 12, 2012 at 2:18 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Apr 12, 2012 1:53 PM, "Ryosuke Niwa" <rniwa at webkit.org> wrote:
> >
> > My suggestion is to try to make it platform-independent but it's
> significantly harder than rebaselining the results for pixel tests. As you
> point out, it defeats one of the benefits of reftests if we end up
> introducing platform-specific reference files.
> >
> > - Ryosuke
> >
>
> It defeats one of the benefits but not all, I think (hope?)
>
Maintaining a reftest with multiple platform-specific baselines seems like
it would be strictly more work than maintaining a pixel test with multiple
platform-specific baselines in most cases.

> -- Dirk
>
> >
> > On Thu, Apr 12, 2012 at 1:46 PM, Jacob Goldstein <jacobg at adobe.com>
> wrote:
> >>
> >> Right, but when those differences come to light, should the goal be to
> adjust the ref test to make it platform-independent, or is having
> platform-specific ref tests acceptable?  Doesn't that put us in the same
> situation as having platform-specific pixel tests?
> >>
> >> From: Ryosuke Niwa <rniwa at webkit.org>
> >> Date: Thu, 12 Apr 2012 13:43:26 -0700
> >> To: Jacob Goldstein <jacobg at adobe.com>
> >> Cc: Ojan Vafai <ojan at chromium.org>, Dirk Pranke <dpranke at chromium.org>,
> WebKit Development <webkit-dev at lists.webkit.org>
> >>
> >> Subject: Re: [webkit-dev] Handling failing reftests
> >>
> >> Right but you wouldn't know platform-specific issues until you land
> them. e.g. rounding errors, subtle font differences in edge cases, etc...
> >>
> >> On Thu, Apr 12, 2012 at 1:30 PM, Jacob Goldstein <jacobg at adobe.com>
> wrote:
> >>>
> >>> Isn't the goal of writing a ref test that it is not platform specific?
>
> >>>
> >>> From: Ryosuke Niwa <rniwa at webkit.org>
> >>> Date: Thu, 12 Apr 2012 12:29:42 -0700
> >>> To: Ojan Vafai <ojan at chromium.org>
> >>> Cc: Dirk Pranke <dpranke at chromium.org>, WebKit Development <
> webkit-dev at lists.webkit.org>
> >>> Subject: Re: [webkit-dev] Handling failing reftests
> >>>
> >>> On Thu, Apr 12, 2012 at 11:53 AM, Ojan Vafai <ojan at chromium.org>
> wrote:
> >>>>
> >>>> I agree that it's hard sometimes to construct reftests that work, but
> once you've done so, the cost on the project of maintaining the test is
> usually considerably lower than pixel tests.
> >>>
> >>>
> >>> Not so sure. There are cases where we have these platform specific
> failures for ref tests, and they're much harder to fix than rebaselining
> pixel results.
> >>>
> >>> - Ryosuke
> >>>
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120412/2403c45b/attachment.html>