[webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

[Darin Adler]

On May 17, 2012, at 4:30 AM, Ojan Vafai wrote:

> I have a proposal that hopefully addresses everyone's concerns, is minimally different from the current format *and* unifies the format with Skipped lists (i.e. Skipped lists as they exist today are valid test_expectations.txt format). The changes from the current format are as follows:
> -Leaving out any "outcomes" means the test is skipped, unless the test has a SLOW modifier, in which case the implied outcome is PASS.
> -Remove the SKIP modifier entirely.
> -Make everything but the test name case-insensitive.
> -Have the test name be the last item on the line
> -Separate modifiers/outcomes/testname with a common delimiter (i.e. ":")
> -Any line starting with // or # is a comment
> -Including a bug entry is optional (maybe only if the test is skipped or wontfix?)
> -Bugs are listed as URLs, except for the bug_ojan format. 
> 
> Examples:
> foo/bar.html # Skipped
> wontfix : foo/bar.html # Skipped and we never intend to fix it. For things like dashboard compatibility tests that only Apple will ever want to make pass
> wontfix : text : foo/bar.html # We never intend to fix this, but we expect it to run and fail text diff. Will still fail if the test times out or crashes.
> webkit.org/b/12345 : text image : foo/bar.html # Flaky. Sometimes only fails text diff, sometimes only fails pixel diff.
> slow mac debug : foo/bar.html # Given extra time to run on mac debug, but is expected to pass.
> image+text : foo/bar.html # Fails both text and pixel diffs
> bug_ojan : fail : foo/bar.html # Fails and ojan is taking responsibility to address the failure.
> bug_ojan : foo/bar.html # Skipped and ojan is taking responsibility to address it.
> 
> # The following would give lint errors
> image : text : foo/bar.html # two outcomes listed in separate groupings
> slow text : foo/bar.html # outcome listed with non-outcome modifier
> crbug.com/12345 text : foo/bar.html # outcome listed with non-outcome modifier
> crbug.com/12345 : wontfix : foo/bar.html # two non-outcomes modifiers listed in separate groupings

Direction seems good.

I’d like to think it through and give some more detailed feedback on some of the specifics, but here are some immediate thoughts.

I don’t understand why we need the ":" or "+" separators. Why not just a list of space-separated words and URLs that can be a mix of modifiers and outcomes? I don’t think we need to support the same words for both modifiers and outcomes. If we want to enforce some kind of order, that would be OK, to reduce arbitrary differences.

Instead of bug_ojan I’d prefer a format more like bug(ojan). I’d like to see a list of the modifier and outcome words so I can easily review them.

Maybe the test specifier (name or directory name) should come first, before the modifiers and outcomes.

-- Darin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/ca1ac7cf/attachment.html>