[webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

[Ojan Vafai]

I have a proposal that hopefully addresses everyone's concerns, is
minimally different from the current format *and* unifies the format with
Skipped lists (i.e. Skipped lists as they exist today are valid
test_expectations.txt format). The changes from the current format are as
follows:
-Leaving out any "outcomes" means the test is skipped, unless the test has
a SLOW modifier, in which case the implied outcome is PASS.
-Remove the SKIP modifier entirely.
-Make everything but the test name case-insensitive.
-Have the test name be the last item on the line
-Separate modifiers/outcomes/testname with a common delimiter (i.e. ":")
-Any line starting with // or # is a comment
-Including a bug entry is optional (maybe only if the test is skipped or
wontfix?)
-Bugs are listed as URLs, except for the bug_ojan format.

Examples:
foo/bar.html # Skipped
wontfix : foo/bar.html # Skipped and we never intend to fix it. For things
like dashboard compatibility tests that only Apple will ever want to make
pass
wontfix : text : foo/bar.html # We never intend to fix this, but we expect
it to run and fail text diff. Will still fail if the test times out or
crashes.
webkit.org/b/12345 : text image : foo/bar.html # Flaky. Sometimes only
fails text diff, sometimes only fails pixel diff.
slow mac debug : foo/bar.html # Given extra time to run on mac debug, but
is expected to pass.
image+text : foo/bar.html # Fails both text and pixel diffs
bug_ojan : fail : foo/bar.html # Fails and ojan is taking responsibility to
address the failure.
bug_ojan : foo/bar.html # Skipped and ojan is taking responsibility to
address it.

# The following would give lint errors
image : text : foo/bar.html # two outcomes listed in separate groupings
slow text : foo/bar.html # outcome listed with non-outcome modifier
crbug.com/12345 text : foo/bar.html # outcome listed with non-outcome
modifier
 crbug.com/12345 : wontfix : foo/bar.html # two non-outcomes modifiers
listed in separate groupings


On Thu, May 17, 2012 at 1:12 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> On Thu, May 17, 2012 at 1:06 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>>  On May 16, 2012, at 10:39 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>> > There was a semi-logical basis, in that the stuff on the right of the
>> > test clearly specified the outcome of the test (PASS, IMAGE, TEXT,
>> > etc.). The stuff on the left was less well defined: there's the bug
>> > numbers, the platform/configuration info (MAC LINUX RELEASE, etc.),
>> > and some other stuff that there is less of a good place for (SLOW,
>> > SKIP, WONTFIX).
>> >
>> > I think it makes sense to syntactically separate at least the
>> > platform/configuration keywords from the outcome keywords. It might be
>> > nice to separate the other things somehow as well, but I don't have
>> > any great ideas for things that are clearly better than the existing
>> > left/right convention.
>>
>> SKIP and WONTFIX seem parallel to PASS to me. I assume TEXT means a
>> failure of text output and IMAGE means a pixel test failure?
>>
>
> Yes.
>
> What does the build configuration info do? Does it apply the line to only
>> those configurations? If that is the case, it does seem potentially
>> different in kind, though maybe also better expressed by being able to
>> combine multiple test_expectations files fro different platform/
>> directories.
>>
>
> Yes. e.g. if you have:
> BUGWK12345 WIN DEBUG : test.html = TEXT
> then we expect test.html to have a failure of text output (i.e.
> -expected.txt doesn't match) on Windows debug builds and expect it to pass
> on all other platforms.
>
> I actually don't like the fact we're sort of duplicating platform fallback
> structure here (e.g. we should be able to have a separate
> text_expectations.txt in platform/chromium-win, platform/chromium-mac,
> etc... to do the same but we don't currently support that) but there was a
> strong resistance to do so among Chromium contributors the last time I
> checked because that'll increase the number of files to maintain.
>
> - Ryosuke
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/555ce684/attachment-0001.html>