[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

[Ryosuke Niwa]

Now that everyone knows the problem, I propose to rename FAIL to DIFF.

FAIL should mean that the test fails, not that it fails with image, text,
or image and text failures.

DIFF, on the other hand, has no ambiguity. It can't be interpreted as
timeout, crash, or pass but can easily be associated with image and text
differences.

- Ryosuke

On Wed, Jun 6, 2012 at 9:42 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> Hi,
>
> As much as it doesn't make any sense to me, FAIL test expectation doesn't
> cover MISSING, CRASH, and TIMEOUT test expectations as it is currently
> interpreted by new-run-webkit-tests. Meaning that
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = FAIL
>
> is logically equivalent to
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = IMAGE TEXT IMAGE+TEXT
>
> Thus, if you're adding a pixel test and only adding the corresponding
> expected result to, say, Mac, then you need to add MISSING results to some
> platforms (e.g. Qt, GTK+, etc...) that cannot find this expected result as
> in:
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = MISSING
>
> If you're expecting the test to fail with image, text, or image and text
> failures, or pass, then do:
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = FAIL PASS
>
> If you're expecting the test to timeout, crash, or, fail (image, text, or
> image and text), or pass, then do:
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = TIMEOUT CRASH FAIL PASS
>
> Best,
> Ryosuke Niwa
> Software Engineer
> Google Inc.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120606/908b94b5/attachment.html>