[webkit-dev] can we stop using Skipped files?

[Dirk Pranke]

On Fri, Jun 8, 2012 at 12:23 PM, Filip Pizlo <fpizlo at apple.com> wrote:
>
> On Jun 8, 2012, at 12:19 PM, Dirk Pranke wrote:
>
>> On Fri, Jun 8, 2012 at 8:21 AM, Filip Pizlo <fpizlo at apple.com> wrote:
>>>
>>> On Jun 8, 2012, at 4:38 AM, Balazs Kelemen <kbalazs at webkit.org> wrote:
>>>
>>>> On 06/08/2012 09:46 AM, Osztrogonac Csaba wrote:
>>>>> Hi,
>>>>>
>>>>> Dirk Pranke írta:
>>>>>> I believe most if not all of the ports have started using either
>>>>>> TestExpectations files or a combination of TestExpectations files
>>>>>> (except for the Apple Win port).
>>>>>>
>>>>>> Can we explicitly switch to the TestExpectations files at this point
>>>>>> and drop support for Skipped files on the other ports (and perhaps
>>>>>> disable old-run-webkit-tests for all but apple win)?
>>>>>
>>>>> Until NRWT can't handle cascaded TestExpectations - https://bugs.webkit.org/show_bug.cgi?id=65834,
>>>>> Qt port can't drop supporting Skipped files. We have many tests skipped in qt-5.0, qt-5.0-wk1,
>>>>> qt-5.0-wk2, wk2 Skipped lists. We can't migrate all of them to the only one TestExpectations.
>>>>>
>>>>> And I disagree with disabling ORWT at all. Qt port still support using ORWT locally.
>>>>> It is better for gardening than NRWT. NRWT regularly has problems with generating
>>>>> new results for a given platform dir (qt,qt-5.0,qt-5.0-wk1,...), it doesn't support
>>>>> the good --skipped=only option . If folks don't want to use it, just not use, but
>>>>> disabling for everyone by fiat isn't a friendly thing.
>>>>
>>>> 1. These are real weaknesses of nrwt, we should fix them. If gardening is better with orwt (i doubt that is the case, but I don't do gardening regularly), we should improve nrwt, i.e. reimplement features from orwt.
>>>
>>> I applaud your enthusiasm.
>>>
>>>> 2. I believe basically everybody agrees that we should drop orwt, except you Ossy. Maybe I'm wrong. So, is there anybody still want to have support for orwt? If so, why?
>>>
>>> I'm with Ossy on this.
>>>
>>> Getting rid of ORWT would be a show stopper for me.
>>
>> Can you file bugs or give me examples of why this is true? Are there
>> specific features missing, or bugs you're hitting?
>
> Here's a really nasty one, that has been sitting untouched since November:
>
> https://bugs.webkit.org/show_bug.cgi?id=71634
>

That bug had fallen off my radar, if it was ever on it (sorry! looks
like this was filed when I was working on chromium side things).

I've now created a tracking bug -
https://bugs.webkit.org/show_bug.cgi?id=88680 - to track all of the
issues I am aware of that we should fix before deleting ORWT. I've
added this, and the bug ossy mentioned about Qt platform baselines,
and the cascading expectations bug.

I am doing another pass over all the open bugs I've seen that are
NRWT-related, and will add more, but if anyone has any others you'd
like to add, please do so.

There is a separate tracking bug -
https://bugs.webkit.org/show_bug.cgi?id=64491 , "polish NRWT until it
shines", that I would prefer to use for any non-show-stopping bugs
(this should not be taken to think those bugs are not important, mind
you).

-- Dirk