[webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

[Adam Barth]

On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting <pkasting at chromium.org>
> wrote:
>> On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger <epoger at chromium.org> wrote:
>>> Can someone please remind me why IMAGE+TEXT even exists?
>>>
>>> Wouldn't it be simpler to just mark a test as follows?
>>>
>>> IMAGE : allow image failure; go red if there is a text failure
>>> TEXT: allow text failure; go red if there is an image failure
>>> IMAGE TEXT: allow text and/or image failure
>>
>> The distinction is that IMAGE TEXT will allow image, text, or both to
>> fail, thus making transitions among the three generate no events.
>>  IMAGE+TEXT says specifically that we expect both to fail and that if one
>> starts passing, someone should do something.  (For example, maybe someone
>> checks in a partial rebaseline where they miss the image expectations.)
>
> Not to bike-shed on anything, but I think we should rename Text and Image to
> TextOnly and ImageOnly. Every single person I know, including myself, had
> never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
> explaining it to him/her .
>
> I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
> checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
> TextExpectations, and it seems that nearly all instances fall into one of
> the two following categories:
>
> 1) text-only test, so IMAGE+TEXT would not have different semantics from
> TEXT (the vast majority)
> 2) Flaky test that may actually pass, so distinguishing what happens with
> the image result is of limited utility (most of these are also text-only
> tests; only a small subset even have an image result)
>
> Thus, I think Fail and ImageOnlyFail would be more useful and understandable
> categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
> semantic that a text failure is expected, and image result if any can either
> pass or fail.

I too would like to see us remove TEXT+IMAGE.  It's really confusing
to non-experts, and it doesn't scale as we introduce new kinds of
failures (like Audio).  Do we really need TEXT+IMAGE+AUDIO,
TEXT+AUDIO, and IMAGE+AUDIO?

Adam