[webkit-dev] generic test expectations?

[Dirk Pranke]

On Tue, Nov 13, 2012 at 12:02 PM, Tony Chang <tony at chromium.org> wrote:
> On Tue, Nov 13, 2012 at 11:35 AM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> On Tue, Nov 13, 2012 at 11:29 AM, Glenn Adams <glenn at skynav.com> wrote:
>> >
>> > On Tue, Nov 13, 2012 at 12:09 PM, Dirk Pranke <dpranke at chromium.org>
>> > wrote:
>> >>
>> >> We don't currently support port-specific reftests (or at least, not
>> >> very well), and we certainly should be trying to minimize where they
>> >> occur.
>> >
>> >
>> > Hmm, I actually used port specific reftest expectation files in a recent
>> > patch [1] (since rolled out), and it appeared to work (as a way to
>> > effectively rebaseline those expectations). So something seems to be
>> > working.
>> >
>> > [1] http://trac.webkit.org/changeset/133529
>> >
>>
>> I expect it'll sort of work, but it won't be robust; you may hit weird
>> behavior and/or bugs. We really haven't beaten on this aspect of
>> things, and I don't know yet how much we want to.
>
>
> I don't think we should support port specific ref test results.  That kind
> of misses the point of using a ref test in the first place.  I mean, you may
> as well check in port specific pixel results which are easier to review for
> correctness.
>
> It may be the case that a ref test is not appropriate for what you're trying
> to test.

I think that there are probably cases where we will have differences
in results because of (legal and entirely correct or permissible)
differences in the implementations and in this case a reftest can
still be an improvement over maintaining N platform-specific pixel
versions.

The obvious example is when we haven't implemented features yet (or have bugs).

The W3C's spec for handling reftests also gives you a way to say "a
test passes if any of these N references may match", which is fairly
consistent with the idea that platform specific references are okay in
some cases.

As to whether pixel-tests are easier to review for correctness than
reftests or not, I think it depends on the test.

-- Dirk