[webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)

[Ojan Vafai]

On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On May 17, 2012, at 1:42 PM, Ojan Vafai <ojan at chromium.org> wrote:
>
> On Thu, May 17, 2012 at 1:37 PM, Peter Kasting <pkasting at chromium.org>wrote:
>
>> On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>
>>> 2. Make outcomes optional. If they are left out, then the test is
>>> skipped (unless the test is marked SLOW, in which case it's expected to
>>> pass). There is no SKIP modifier.
>>>
>>
>> I don't think we should do this.  It seems very subtle.  I'd rather be
>> explicit.
>>
>> I'm OK with the rest of your numbered proposals.
>>
>
> I disagree, but I'm fine with punting this to the list of controversial
> changes that we should discuss separately. FWIW, my main motivation here is
> that it allows us to unify the Skipped file format with the
> test_expectations.txt format. But again, we can discuss that separately.
>
>
> Adding SKIP (or whatever) to every line of skipped files is not a big
> hurdle, I think we could live with that is a transitions tep. I think the
> bigger hurdle is supporting chaining across multiple directories.
>

That's great. I don't think anyone is opposed to adding chaining and I
think that's on Dirk short-list of todos.

The only potentially tricky thing here is figuring out what the platform
modifiers mean for non-Chromium ports, e.g. I imagine Qt will want similar
modifiers to Chromium (mac, linux, win, debug, release, etc). But I think
the difficulty here is more in getting the python code right than agreeing
on what the correct behavior is.

Also, currently the test_expectations.txt format requires either a bug
number or a bug(ojan) entry. Would that be OK with you too? It has proven
really good historically for keeping track of why a test was added to the
file and for keeping track of getting the tests fixed (or, more
importantly, having someone responsible for following up on it), but we
could easily restrict this requirement to the Chromium expectations file if
other ports dislike it.

I think with those three things and
https://bugs.webkit.org/show_bug.cgi?id=86796 addressed, then the formats
will be unified and the only thing to bikeshed over is the filename. :)

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/62a869f4/attachment.html>