[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

[Peter Kasting]

On Wed, Jun 6, 2012 at 10:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> Now that everyone knows the problem, I propose to rename FAIL to DIFF.
>
> FAIL should mean that the test fails, not that it fails with image, text,
> or image and text failures.
>
> DIFF, on the other hand, has no ambiguity. It can't be interpreted as
> timeout, crash, or pass but can easily be associated with image and text
> differences.
>

I don't think DIFF is any better.  It sounds like it means the output is
"different than what we wanted", thus it effectively means "didn't pass",
and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would
expect FAIL to.

Personally I'd prefer to resolve this -- if we need to -- by removing FAIL
entirely.  Being explicit about your expectations isn't a bad thing.  Plus,
the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems likely to be
small.

PK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120606/edeeb244/attachment.html>