[webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)

[Ojan Vafai]

On Thu, May 17, 2012 at 10:37 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On May 17, 2012, at 7:27 PM, Ojan Vafai <ojan at chromium.org> wrote:
>
> On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>>
>> On May 17, 2012, at 1:42 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>
>> On Thu, May 17, 2012 at 1:37 PM, Peter Kasting <pkasting at chromium.org>wrote:
>>
>>> On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>>
>>>> 2. Make outcomes optional. If they are left out, then the test is
>>>> skipped (unless the test is marked SLOW, in which case it's expected to
>>>> pass). There is no SKIP modifier.
>>>>
>>>
>>> I don't think we should do this.  It seems very subtle.  I'd rather be
>>> explicit.
>>>
>>> I'm OK with the rest of your numbered proposals.
>>>
>>
>> I disagree, but I'm fine with punting this to the list of controversial
>> changes that we should discuss separately. FWIW, my main motivation here is
>> that it allows us to unify the Skipped file format with the
>> test_expectations.txt format. But again, we can discuss that separately.
>>
>>
>> Adding SKIP (or whatever) to every line of skipped files is not a big
>> hurdle, I think we could live with that is a transitions tep. I think the
>> bigger hurdle is supporting chaining across multiple directories.
>>
>
> That's great. I don't think anyone is opposed to adding chaining and I
> think that's on Dirk short-list of todos.
>
> The only potentially tricky thing here is figuring out what the platform
> modifiers mean for non-Chromium ports, e.g. I imagine Qt will want similar
> modifiers to Chromium (mac, linux, win, debug, release, etc). But I think
> the difficulty here is more in getting the python code right than agreeing
> on what the correct behavior is.
>
>
> I think it would be good if platform modifiers in the expectations file
> matched the platform names we use under the platform/ directory, either
> literally or as a suffix. So for example "mac" in the chromium expectations
> file could mean chromium-mac, in the qt expectations file it could mean
> qt-mac, in the mac expectations file it should not be used, but snowleopard
> would mean mac-snowleopard.
>

That seems like a good way to organize it to me. Dirk, that make sense to
you?

Implementation detail: this might be nice for automating the generation of
the macros used for determining what the generic ones map to (e.g. the mac
maps to snowleopard, leopard, mt lion for chromium). It would be a nice
secondary benefit to get rid of the current hard-coded lists. Only thing
we'd need to hard code is what platform the generic directory maps to (e.g.
that chromium-mac is actually the chromium mt lion directory).

> Also, currently the test_expectations.txt format requires either a bug
> number or a bug(ojan) entry. Would that be OK with you too? It has proven
> really good historically for keeping track of why a test was added to the
> file and for keeping track of getting the tests fixed (or, more
> importantly, having someone responsible for following up on it), but we
> could easily restrict this requirement to the Chromium expectations file if
> other ports dislike it.
>
> Requiring a bug seems good. I don't personally see the need for any
> exception to having a filed and tracked bug but perhaps folks closer to the
> problem know of a reason.
>

Great. That's simpler.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120517/4c5b6de8/attachment.html>