[webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

[Ojan Vafai]

Closing the loop...bug filed
https://bugs.webkit.org/show_bug.cgi?id=86796 minus
item 2 since that turned out to be controversial.

On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan at chromium.org> wrote:

> We are arguing about too many orthogonal things at once. It seems like
> there are a few things we can agree on. Lets make incremental progress on
> those and after have a more targeted discussion on the controversial issues
> that are left. We don't need to make this change in one big commit (and
> shouldn't really).
>
> 1. Make SLOW and WONTFIX outcomes instead of modifiers.
> 2. Make outcomes optional. If they are left out, then the test is skipped
> (unless the test is marked SLOW, in which case it's expected to pass).
> There is no SKIP modifier.
> 3. Change bug modifiers to be URLs. The format for person bugs is
> bug(ojan).
> 4. The only things that come before the test are bug modifiers and
> platform/configuration modifiers (e.g. MAC DEBUG).
> 5. WONTFIX tests are either skipped or we ignore their output as long as
> they don't crash (I don't care which one we do).
> 6. Use the same character as a separator before and after the test (":").
> 7. Standardize both Skipped and test_expectations.txt on a common comment
> delimiter (i.e. "#").
>
> Examples:
> webkit.org/b/12345 : foo/bar.html # test is skipped on all
> platforms/configurations this test_expectations.txt file applies to
> MAC DEBUG : foo/bar.html : SLOW # test is slow on mac debug, but is
> expected to pass
> crbug.com/1234 : foo/bar.html : SLOW TEXT IMAGE # test is slow and flaky.
> sometimes fails text diff, other times fails image diff
>
> Talking to people off-thread, there are strong differing opinions on the
> below controversial issues, so we should discuss separately since getting
> consensus will be difficult (maybe even on dedicated email threads):
> -Lowercasing the modifiers/outcomes
> -Moving all modifiers/outcomes before/after the test name
> -Making bug modifiers optional for non-wontfix tests
> -Getting rid of delimiters entirely
>
>
> On Thu, May 17, 2012 at 12:53 PM, Dirk Pranke <dpranke at chromium.org>wrote:
>
>> On Thu, May 17, 2012 at 12:47 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> > On Thu, May 17, 2012 at 12:36 PM, Dirk Pranke <dpranke at chromium.org>
>> wrote:
>> >>
>> >> On Thu, May 17, 2012 at 4:30 AM, Ojan Vafai <ojan at chromium.org> wrote:
>> >> > -Make everything but the test name case-insensitive.
>> >>
>> >> I don't think I like this; it could lead to a lot of arbitrarily
>> >> different formatting in the file, making things harder to read.
>> >
>> >
>> > Modifiers and expectations are already case-insensitive as far as I
>> read the
>> > code yesterday.
>> >
>>
>> It may be that it's legal to mix the case, but no one does it.
>>
>> >> I think we'd probably find ourselves (re-)converging on some convention
>> >> pretty quickly. I personally find all uppercase fairly easy to read in
>> >> this case since it distinguishes the modifiers from the test name.
>> >
>> >
>> > I think this problem will disappear once we place modifiers and
>> expectations
>> > on the same side
>> > because then there is exactly one place those tokens could appear.
>> There is
>> > no need to scan
>> > through a line then.
>> >
>>
>> It's possible. As I said, having some other clear delimiter would help.
>>
>> >>
>> >> If we have some other clear delimiter this would probably be less
>> >> important, in which case all lower case would be fine as well. Initial
>> caps
>> >> seems less good to me.
>> >
>> >
>> > I find either all-lowercase or all-caps to be much harder to read than
>> > capitalized words. They look like a blob of letters to me.
>>
>> We might have to agree to disagree here, then, but that's fine.
>>
>> If there was a clear consensus that one style or another is better, we
>> should go with that.
>>
>> > Also, I don't
>> > think we use all-caps name anywhere else in WebKit so it's inconsistent
>> with
>> > the convention we use elsewhere.
>> >
>>
>> I don't think this particularly matters. We should design a format
>> based on what is most useful in this context.
>>
>> -- Dirk
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/717a6de5/attachment.html>