[webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling "failing" layout tests and TestExpectations

[Dirk Pranke]

On Fri, Aug 17, 2012 at 5:01 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Fri, Aug 17, 2012 at 4:55 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>
>> Asserting a test case is 100% correct is nearly impossible for a large
>> percentage of tests. The main advantage it gives us is the ability to have
>> -expected mean "unsure".
>>
>> Lets instead only add -failing (i.e. no -passing). Leaving -expected to
>> mean roughly what it does today to Chromium folk (roughly, as best we can
>> tell this test is passing). -failing means it's *probably* an incorrect
>> result but needs an expert to look at it to either mark it correct (i.e.
>> rename it to -expected) or figure out how the root cause of the bug.
>>
>> This actually matches exactly what Chromium gardeners do today, except
>> instead of putting a line in TestExpectations/Skipped to look at later, they
>> checkin the -failing file to look at later, which has all the advantages
>> Dirk listed in the other thread.
>
>
> I'm much more comfortable with this proposal.
>

While this obviously isn't my full proposal, it's at least a step down
the path, so I'm fine with it.

To Filip and Ryosuke's points, I am concurrently working to reduce the
other complexity we have (e.g., moving everyone to TestExpectations,
getting the new syntax implemented, etc) as well.

-- Dirk