[webkit-dev] PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT

[Dirk Pranke]

The reason for this (which is debatable) is that CRASH and TIMEOUT are
deemed to be more serious and shouldn't be suppressed as lightly.
MISSING, on the other hand, just indicates that there's something
wrong (tests should never have missing results for very long).

-- Dirk

On Wed, Jun 6, 2012 at 9:42 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> Hi,
>
> As much as it doesn't make any sense to me, FAIL test expectation doesn't
> cover MISSING, CRASH, and TIMEOUT test expectations as it is currently
> interpreted by new-run-webkit-tests. Meaning that
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = FAIL
>
> is logically equivalent to
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = IMAGE TEXT IMAGE+TEXT
>
> Thus, if you're adding a pixel test and only adding the corresponding
> expected result to, say, Mac, then you need to add MISSING results to some
> platforms (e.g. Qt, GTK+, etc...) that cannot find this expected result as
> in:
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = MISSING
>
> If you're expecting the test to fail with image, text, or image and text
> failures, or pass, then do:
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = FAIL PASS
>
> If you're expecting the test to timeout, crash, or, fail (image, text, or
> image and text), or pass, then do:
>
> BUGRNIWA : some-pixel-test-i-am-adding.html = TIMEOUT CRASH FAIL PASS
>
> Best,
> Ryosuke Niwa
> Software Engineer
> Google Inc.
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>