[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

Peter Kasting pkasting at chromium.org
Wed Jun 6 22:36:25 PDT 2012

On Wed, Jun 6, 2012 at 10:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> Now that everyone knows the problem, I propose to rename FAIL to DIFF.
> FAIL should mean that the test fails, not that it fails with image, text,
> or image and text failures.
> DIFF, on the other hand, has no ambiguity. It can't be interpreted as
> timeout, crash, or pass but can easily be associated with image and text
> differences.

I don't think DIFF is any better.  It sounds like it means the output is
"different than what we wanted", thus it effectively means "didn't pass",
and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would
expect FAIL to.

Personally I'd prefer to resolve this -- if we need to -- by removing FAIL
entirely.  Being explicit about your expectations isn't a bad thing.  Plus,
the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems likely to be

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120606/edeeb244/attachment.html>

More information about the webkit-dev mailing list