[webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

[Dirk Pranke]

On Wed, Jun 13, 2012 at 4:38 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> > On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain <benjamin at webkit.org>
>> > wrote:
>> >> On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke <dpranke at chromium.org>
>> >> wrote:
>> >> > webkit.org/12345 WIN MAC DEBUG \
>> >> > animations/stop-animation-on-suspend.html \ CRASH TEXT PASS
>> >>
>> >> My bikeshedding:
>> >> -In my opinion, the backslash is not any better than the old
>> >> delimiter. It looks like we are escaping something.
>> >>
>> >> I liked better the proposal to have platform and results grouped
>> >> instead of the delimiter.
>> >> e.g: webkit.org/12345 (WIN MAC DEBUG)
>> >> animations/stop-animation-on-suspend.html (CRASH TEXT PASS)
>> >>
>> >> -I prefer lowercase keywords, possibly with the first character
>> >> uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all
>> >> uppercase keywords somewhere else in the project?
>> >
>> > So something like
>> > webkit.org/12345 [Win Mac
>> > Debug] animations/stop-animation-on-suspend.html
>> > [Crash Text Pass]
>> > ? I like that!
>>
>> This seems quite readable to me as well. As mentioned above, I'd be
>> inclined to make the absence of [] on the right hand side mean "SKIP".
>
>
> That sounds very reasonable.
>
> In fact, I'd argue that it should be the only way to skip tests. It makes no
> sense to have any other test expectation when a test is skipped because the
> types of failures of skipped tests (when actually ran) will very likely
> change over time, and there is no way to track them.
>

There's still a distinction between SKIP (temporary) and WONTFIX
(permanent until some plan changes) that we should preserve, I think.
It could be useful to indicate SKIP CRASH or SKIP TIMEOUT as well, but
I agree that the type of failure will probably change over time.

-- Dirk