[webkit-dev] Handling failing reftests

[Ojan Vafai]

Ossy, I think the right thing to do, in general, with failing reftests is
to skip them.

Dave, you can see the chromium win/linux failures here (it passes on
chromium mac):
http://test-results.appspot.com/dashboards/flakiness_dashboard.html#showExpectations=true&showLargeExpectations=true&revision=113914&tests=fast%2Fmulticol%2Fcell-shrinkback.html.
Search the page for "DIFF:" to see the image diffs.

I agree that it's hard sometimes to construct reftests that work, but once
you've done so, the cost on the project of maintaining the test is usually
considerably lower than pixel tests.

On Thu, Apr 12, 2012 at 11:23 AM, David Hyatt <hyatt at apple.com> wrote:

> That reftest is mine. It was supposed to work cross-platform. I am happy
> to fix, but I'd need to understand what about it didn't work on other
> platforms. Can someone send me a screenshot of what it looks like?
>
> (This is actually why I hate reftests for pagination. It's very hard to
> construct an unpaginated version that matches the paginated layout without
> tripping over stuff like this.)
>
> Thanks
> Dave
> (hyatt at apple.com)
>
> On Apr 12, 2012, at 1:01 PM, Ryosuke Niwa wrote:
>
> On Thu, Apr 12, 2012 at 10:57 AM, Dirk Pranke <dpranke at chromium.org>wrote:
>>
>> This is an interesting situation; does it make sense to require our
>> reftests to be generic, or is that unrealistic? I also wonder if we
>> should allow  platform-specific pixel tests to coexist with
>> platform-specific reference tests (since you could obviously write a
>> platform-specific ref test that just displayed a single IMG, but the
>> tools don't make that particularly easy).
>>
>
> Allowing platform-specific pixel results for a ref test is a bad idea IMO.
> It makes things even more complicated and harder to understand than it
> already is today.
>
> - Ryosuke
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120412/023c94a9/attachment.html>