[webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling "failing" layout tests and TestExpectations

[Filip Pizlo]

Then I am on board.

We still do need to revisit the handling of flaky tests.  The current approach is an absolute disaster.  (I normally love exaggerating, but in this case, I feel no satisfaction in doing so because it is at best an understatement.)

-Filip



On Aug 17, 2012, at 5:36 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> All non-flaky failures, yes.
> 
> Flaky tests would still require entries in the TestExpectations files
> at this time; discussion of how to treat them is a separate topic.
> 
> -- Dirk
> 
> On Fri, Aug 17, 2012 at 5:35 PM, Filip Pizlo <fpizlo at apple.com> wrote:
>> +1, contingent upon the following: are we agreeing that all current uses of
>> TEXT, IMAGE, and so forth in TestExpectations should be in the *very near
>> term* following Dirk's change be turned into -failing files?
>> 
>> -Filip
>> 
>> 
>> On Aug 17, 2012, at 5:01 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> 
>> On Fri, Aug 17, 2012 at 4:55 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>> 
>>> Asserting a test case is 100% correct is nearly impossible for a large
>>> percentage of tests. The main advantage it gives us is the ability to have
>>> -expected mean "unsure".
>>> 
>>> Lets instead only add -failing (i.e. no -passing). Leaving -expected to
>>> mean roughly what it does today to Chromium folk (roughly, as best we can
>>> tell this test is passing). -failing means it's *probably* an incorrect
>>> result but needs an expert to look at it to either mark it correct (i.e.
>>> rename it to -expected) or figure out how the root cause of the bug.
>>> 
>>> This actually matches exactly what Chromium gardeners do today, except
>>> instead of putting a line in TestExpectations/Skipped to look at later, they
>>> checkin the -failing file to look at later, which has all the advantages
>>> Dirk listed in the other thread.
>> 
>> 
>> I'm much more comfortable with this proposal.
>> 
>> - Ryosuke
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo/webkit-dev
>> 
>>