[webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

[Ryosuke Niwa]

On Thu, May 17, 2012 at 1:06 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>  On May 16, 2012, at 10:39 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> > There was a semi-logical basis, in that the stuff on the right of the
> > test clearly specified the outcome of the test (PASS, IMAGE, TEXT,
> > etc.). The stuff on the left was less well defined: there's the bug
> > numbers, the platform/configuration info (MAC LINUX RELEASE, etc.),
> > and some other stuff that there is less of a good place for (SLOW,
> > SKIP, WONTFIX).
> >
> > I think it makes sense to syntactically separate at least the
> > platform/configuration keywords from the outcome keywords. It might be
> > nice to separate the other things somehow as well, but I don't have
> > any great ideas for things that are clearly better than the existing
> > left/right convention.
>
> SKIP and WONTFIX seem parallel to PASS to me. I assume TEXT means a
> failure of text output and IMAGE means a pixel test failure?
>

Yes.

What does the build configuration info do? Does it apply the line to only
> those configurations? If that is the case, it does seem potentially
> different in kind, though maybe also better expressed by being able to
> combine multiple test_expectations files fro different platform/
> directories.
>

Yes. e.g. if you have:
BUGWK12345 WIN DEBUG : test.html = TEXT
then we expect test.html to have a failure of text output (i.e.
-expected.txt doesn't match) on Windows debug builds and expect it to pass
on all other platforms.

I actually don't like the fact we're sort of duplicating platform fallback
structure here (e.g. we should be able to have a separate
text_expectations.txt in platform/chromium-win, platform/chromium-mac,
etc... to do the same but we don't currently support that) but there was a
strong resistance to do so among Chromium contributors the last time I
checked because that'll increase the number of files to maintain.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/dedd0f9a/attachment.html>