[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

[Dirk Pranke]

On Wed, Jun 6, 2012 at 10:36 PM, Peter Kasting <pkasting at chromium.org> wrote:
> On Wed, Jun 6, 2012 at 10:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>
>> Now that everyone knows the problem, I propose to rename FAIL to DIFF.
>>
>> FAIL should mean that the test fails, not that it fails with image, text,
>> or image and text failures.
>>
>> DIFF, on the other hand, has no ambiguity. It can't be interpreted as
>> timeout, crash, or pass but can easily be associated with image and text
>> differences.
>
>
> I don't think DIFF is any better.  It sounds like it means the output is
> "different than what we wanted", thus it effectively means "didn't pass",
> and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would
> expect FAIL to.
>
> Personally I'd prefer to resolve this -- if we need to -- by removing FAIL
> entirely.  Being explicit about your expectations isn't a bad thing.  Plus,
> the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems likely to be
> small.

FAIL was supposed to have been removed long ago in favor of
TEXT/IMAGE/IMAGE+TEXT as necessary as Peter suggests, but I never got
around to it :) I agree with his reasoning, and have not been
convinced that not having to be specific about what is failing is a
good thing.

I also agree that we shouldn't TEXT/IMAGE/IMAGE+TEXT in the file at
all for extended periods of time since you can just check in updated
baselines.

-- Dirk