[webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)

[Ryosuke Niwa]

On Thu, May 17, 2012 at 7:27 PM, Ojan Vafai <ojan at chromium.org> wrote:

> On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>> On May 17, 2012, at 1:42 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>
>> On Thu, May 17, 2012 at 1:37 PM, Peter Kasting <pkasting at chromium.org>wrote:
>>
>>> On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>>
>>>> 2. Make outcomes optional. If they are left out, then the test is
>>>> skipped (unless the test is marked SLOW, in which case it's expected to
>>>> pass). There is no SKIP modifier.
>>>>
>>>
>>> I don't think we should do this.  It seems very subtle.  I'd rather be
>>> explicit.
>>>
>>> I'm OK with the rest of your numbered proposals.
>>>
>>
>> I disagree, but I'm fine with punting this to the list of controversial
>> changes that we should discuss separately. FWIW, my main motivation here is
>> that it allows us to unify the Skipped file format with the
>> test_expectations.txt format. But again, we can discuss that separately.
>>
>>
>> Adding SKIP (or whatever) to every line of skipped files is not a big
>> hurdle, I think we could live with that is a transitions tep. I think the
>> bigger hurdle is supporting chaining across multiple directories.
>>
>
> That's great. I don't think anyone is opposed to adding chaining and I
> think that's on Dirk short-list of todos.
>
> The only potentially tricky thing here is figuring out what the platform
> modifiers mean for non-Chromium ports, e.g. I imagine Qt will want similar
> modifiers to Chromium (mac, linux, win, debug, release, etc). But I think
> the difficulty here is more in getting the python code right than agreeing
> on what the correct behavior is.
>
> Also, currently the test_expectations.txt format requires either a bug
> number or a bug(ojan) entry. Would that be OK with you too? It has proven
> really good historically for keeping track of why a test was added to the
> file and for keeping track of getting the tests fixed (or, more
> importantly, having someone responsible for following up on it), but we
> could easily restrict this requirement to the Chromium expectations file if
> other ports dislike it.
>

Most of entries in Skipped files have a bug URL associated with them so
this is probably not a problem.

I can see that we probably want to omit it for WontFix or NotImplemented
since there isn't really a point in filing a bug for WontFix features but
we already do that so there should be no blocker.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/f4315aa6/attachment.html>