[webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

[Ryosuke Niwa]

On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth <abarth at webkit.org> wrote:

> On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> > On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> > On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting <pkasting at chromium.org>
> > wrote:
> >> On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger <epoger at chromium.org>
> wrote:
> >>> Can someone please remind me why IMAGE+TEXT even exists?
> >>>
> >>> Wouldn't it be simpler to just mark a test as follows?
> >>>
> >>> IMAGE : allow image failure; go red if there is a text failure
> >>> TEXT: allow text failure; go red if there is an image failure
> >>> IMAGE TEXT: allow text and/or image failure
> >>
> >> The distinction is that IMAGE TEXT will allow image, text, or both to
> >> fail, thus making transitions among the three generate no events.
> >>  IMAGE+TEXT says specifically that we expect both to fail and that if
> one
> >> starts passing, someone should do something.  (For example, maybe
> someone
> >> checks in a partial rebaseline where they miss the image expectations.)
> >
> > Not to bike-shed on anything, but I think we should rename Text and
> Image to
> > TextOnly and ImageOnly. Every single person I know, including myself, had
> > never got the distinction between IMAGE TEXT and IMAGE+TEXT without
> someone
> > explaining it to him/her .
> >
> > I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
> > checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
> > TextExpectations, and it seems that nearly all instances fall into one of
> > the two following categories:
> >
> > 1) text-only test, so IMAGE+TEXT would not have different semantics from
> > TEXT (the vast majority)
> > 2) Flaky test that may actually pass, so distinguishing what happens with
> > the image result is of limited utility (most of these are also text-only
> > tests; only a small subset even have an image result)
> >
> > Thus, I think Fail and ImageOnlyFail would be more useful and
> understandable
> > categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have
> the
> > semantic that a text failure is expected, and image result if any can
> either
> > pass or fail.
>
> I too would like to see us remove TEXT+IMAGE.  It's really confusing
> to non-experts, and it doesn't scale as we introduce new kinds of
> failures (like Audio).  Do we really need TEXT+IMAGE+AUDIO,
> TEXT+AUDIO, and IMAGE+AUDIO?
>

+1 to that. Also, I can never remember whether it's IMAGE+TEXT or
TEXT+IMAGE (it's IMAGE+TEXT).

But I agree with Dirk that we should probably discuss about this on a
separate thread.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120614/aed963b1/attachment.html>