[webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

[Dimitri Glazkov]

On Thu, May 17, 2012 at 9:11 AM, Darin Adler <darin at apple.com> wrote:
> On May 17, 2012, at 4:30 AM, Ojan Vafai wrote:
>
> I have a proposal that hopefully addresses everyone's concerns, is minimally
> different from the current format *and* unifies the format with Skipped
> lists (i.e. Skipped lists as they exist today are valid
> test_expectations.txt format). The changes from the current format are as
> follows:
> -Leaving out any "outcomes" means the test is skipped, unless the test has a
> SLOW modifier, in which case the implied outcome is PASS.
> -Remove the SKIP modifier entirely.
> -Make everything but the test name case-insensitive.
> -Have the test name be the last item on the line
> -Separate modifiers/outcomes/testname with a common delimiter (i.e. ":")
> -Any line starting with // or # is a comment
> -Including a bug entry is optional (maybe only if the test is skipped or
> wontfix?)
> -Bugs are listed as URLs, except for the bug_ojan format.
>
> Examples:
> foo/bar.html # Skipped
> wontfix : foo/bar.html # Skipped and we never intend to fix it. For things
> like dashboard compatibility tests that only Apple will ever want to make
> pass
> wontfix : text : foo/bar.html # We never intend to fix this, but we expect
> it to run and fail text diff. Will still fail if the test times out or
> crashes.
> webkit.org/b/12345 : text image : foo/bar.html # Flaky. Sometimes only fails
> text diff, sometimes only fails pixel diff.
> slow mac debug : foo/bar.html # Given extra time to run on mac debug, but is
> expected to pass.
> image+text : foo/bar.html # Fails both text and pixel diffs
> bug_ojan : fail : foo/bar.html # Fails and ojan is taking responsibility to
> address the failure.
> bug_ojan : foo/bar.html # Skipped and ojan is taking responsibility to
> address it.
>
> # The following would give lint errors
> image : text : foo/bar.html # two outcomes listed in separate groupings
> slow text : foo/bar.html # outcome listed with non-outcome modifier
> crbug.com/12345 text : foo/bar.html # outcome listed with non-outcome
> modifier
> crbug.com/12345 : wontfix : foo/bar.html # two non-outcomes modifiers listed
> in separate groupings
>
>
> Direction seems good.
>
> I’d like to think it through and give some more detailed feedback on some of
> the specifics, but here are some immediate thoughts.
>
> I don’t understand why we need the ":" or "+" separators. Why not just a
> list of space-separated words and URLs that can be a mix of modifiers and
> outcomes? I don’t think we need to support the same words for both modifiers
> and outcomes. If we want to enforce some kind of order, that would be OK, to
> reduce arbitrary differences.

Maybe it's the Stockholm syndrome talking, but I actually quite like
the clear delineation between test modifiers and test expectations. My
intuitive reaction to the attempts to bunch them all together is that
it will make both writing and reading of a test expectation harder.
Having them on either side of the test file also helps, since the test
file functions as a natural delimiter. I certainly don't want to be
trying to recall who's-on-firsts of modifiers and expectations, if
they are on the same side of the file.

Sure, there are some bugs in existing formats (like SLOW being a
modifier), but can we just fix those and avoid boiling the ocean? :)

:DG<

>
> Instead of bug_ojan I’d prefer a format more like bug(ojan). I’d like to see
> a list of the modifier and outcome words so I can easily review them.
>
> Maybe the test specifier (name or directory name) should come first, before
> the modifiers and outcomes.
>
> -- Darin
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>