[webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

[Maciej Stachowiak]

On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> 
> On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting <pkasting at chromium.org> wrote:
> On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger <epoger at chromium.org> wrote:
> Can someone please remind me why IMAGE+TEXT even exists?
> 
> Wouldn't it be simpler to just mark a test as follows?
> IMAGE : allow image failure; go red if there is a text failure
> TEXT: allow text failure; go red if there is an image failure
> IMAGE TEXT: allow text and/or image failure
> The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events.  IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something.  (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.)
> 
> Not to bike-shed on anything, but I think we should rename Text and Image to TextOnly and ImageOnly. Every single person I know, including myself, had never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone explaining it to him/her .

I think IMAGE+TEXT is not a very useful distinction from TEXT either. I checked for uses of TEXT that is not IMAGE+TEXT in the Chromium TextExpectations, and it seems that nearly all instances fall into one of the two following categories:

1) text-only test, so IMAGE+TEXT would not have different semantics from TEXT (the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with the image result is of limited utility (most of these are also text-only tests; only a small subset even have an image result)

Thus, I think Fail and ImageOnlyFail would be more useful and understandable categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the semantic that a text failure is expected, and image result if any can either pass or fail.

Regards,
Maciej

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120614/0d7593c6/attachment.html>