[webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)

[Maciej Stachowiak]

On May 17, 2012, at 7:27 PM, Ojan Vafai <ojan at chromium.org> wrote:

> On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> 
> On May 17, 2012, at 1:42 PM, Ojan Vafai <ojan at chromium.org> wrote:
> 
>> On Thu, May 17, 2012 at 1:37 PM, Peter Kasting <pkasting at chromium.org> wrote:
>> On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan at chromium.org> wrote:
>> 2. Make outcomes optional. If they are left out, then the test is skipped (unless the test is marked SLOW, in which case it's expected to pass). There is no SKIP modifier.
>> 
>> I don't think we should do this.  It seems very subtle.  I'd rather be explicit.
>> 
>> I'm OK with the rest of your numbered proposals.
>> 
>> I disagree, but I'm fine with punting this to the list of controversial changes that we should discuss separately. FWIW, my main motivation here is that it allows us to unify the Skipped file format with the test_expectations.txt format. But again, we can discuss that separately.
> 
> Adding SKIP (or whatever) to every line of skipped files is not a big hurdle, I think we could live with that is a transitions tep. I think the bigger hurdle is supporting chaining across multiple directories.
> 
> That's great. I don't think anyone is opposed to adding chaining and I think that's on Dirk short-list of todos.
> 
> The only potentially tricky thing here is figuring out what the platform modifiers mean for non-Chromium ports, e.g. I imagine Qt will want similar modifiers to Chromium (mac, linux, win, debug, release, etc). But I think the difficulty here is more in getting the python code right than agreeing on what the correct behavior is.

I think it would be good if platform modifiers in the expectations file matched the platform names we use under the platform/ directory, either literally or as a suffix. So for example "mac" in the chromium expectations file could mean chromium-mac, in the qt expectations file it could mean qt-mac, in the mac expectations file it should not be used, but snowleopard would mean mac-snowleopard.

> 
> Also, currently the test_expectations.txt format requires either a bug number or a bug(ojan) entry. Would that be OK with you too? It has proven really good historically for keeping track of why a test was added to the file and for keeping track of getting the tests fixed (or, more importantly, having someone responsible for following up on it), but we could easily restrict this requirement to the Chromium expectations file if other ports dislike it.

Requiring a bug seems good. I don't personally see the need for any exception to having a filed and tracked bug but perhaps folks closer to the problem know of a reason.

> 
> I think with those three things and https://bugs.webkit.org/show_bug.cgi?id=86796 addressed, then the formats will be unified and the only thing to bikeshed over is the filename. :)

And maybe more follow-up discussion about the syntax.

Regards,
Maciej


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/168d849d/attachment.html>