[webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

[Ojan Vafai]

We are arguing about too many orthogonal things at once. It seems like
there are a few things we can agree on. Lets make incremental progress on
those and after have a more targeted discussion on the controversial issues
that are left. We don't need to make this change in one big commit (and
shouldn't really).

1. Make SLOW and WONTFIX outcomes instead of modifiers.
2. Make outcomes optional. If they are left out, then the test is skipped
(unless the test is marked SLOW, in which case it's expected to pass).
There is no SKIP modifier.
3. Change bug modifiers to be URLs. The format for person bugs is bug(ojan).
4. The only things that come before the test are bug modifiers and
platform/configuration modifiers (e.g. MAC DEBUG).
5. WONTFIX tests are either skipped or we ignore their output as long as
they don't crash (I don't care which one we do).
6. Use the same character as a separator before and after the test (":").
7. Standardize both Skipped and test_expectations.txt on a common comment
delimiter (i.e. "#").

Examples:
webkit.org/b/12345 : foo/bar.html # test is skipped on all
platforms/configurations this test_expectations.txt file applies to
MAC DEBUG : foo/bar.html : SLOW # test is slow on mac debug, but is
expected to pass
crbug.com/1234 : foo/bar.html : SLOW TEXT IMAGE # test is slow and flaky.
sometimes fails text diff, other times fails image diff

Talking to people off-thread, there are strong differing opinions on the
below controversial issues, so we should discuss separately since getting
consensus will be difficult (maybe even on dedicated email threads):
-Lowercasing the modifiers/outcomes
-Moving all modifiers/outcomes before/after the test name
-Making bug modifiers optional for non-wontfix tests
-Getting rid of delimiters entirely


On Thu, May 17, 2012 at 12:53 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Thu, May 17, 2012 at 12:47 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> > On Thu, May 17, 2012 at 12:36 PM, Dirk Pranke <dpranke at chromium.org>
> wrote:
> >>
> >> On Thu, May 17, 2012 at 4:30 AM, Ojan Vafai <ojan at chromium.org> wrote:
> >> > -Make everything but the test name case-insensitive.
> >>
> >> I don't think I like this; it could lead to a lot of arbitrarily
> >> different formatting in the file, making things harder to read.
> >
> >
> > Modifiers and expectations are already case-insensitive as far as I read
> the
> > code yesterday.
> >
>
> It may be that it's legal to mix the case, but no one does it.
>
> >> I think we'd probably find ourselves (re-)converging on some convention
> >> pretty quickly. I personally find all uppercase fairly easy to read in
> >> this case since it distinguishes the modifiers from the test name.
> >
> >
> > I think this problem will disappear once we place modifiers and
> expectations
> > on the same side
> > because then there is exactly one place those tokens could appear. There
> is
> > no need to scan
> > through a line then.
> >
>
> It's possible. As I said, having some other clear delimiter would help.
>
> >>
> >> If we have some other clear delimiter this would probably be less
> >> important, in which case all lower case would be fine as well. Initial
> caps
> >> seems less good to me.
> >
> >
> > I find either all-lowercase or all-caps to be much harder to read than
> > capitalized words. They look like a blob of letters to me.
>
> We might have to agree to disagree here, then, but that's fine.
>
> If there was a clear consensus that one style or another is better, we
> should go with that.
>
> > Also, I don't
> > think we use all-caps name anywhere else in WebKit so it's inconsistent
> with
> > the convention we use elsewhere.
> >
>
> I don't think this particularly matters. We should design a format
> based on what is most useful in this context.
>
> -- Dirk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/7ecba5fa/attachment.html>