[webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling "failing" layout tests and TestExpectations

[Dirk Pranke]

On Tue, Aug 21, 2012 at 4:16 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
> On Aug 21, 2012, at 3:23 PM, Ojan Vafai <ojan at chromium.org> wrote:
>
> On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>> Here's how I imagine the workflow when a sheriff or just innocent
>> bystander notices a deterministically failing test. Follow this two-step
>> algorithm:
>>
>> 1) Are you confident that the new result is an improvement or no worse? If
>> so, then simply update -expected.txt.
>> 2) Otherwise, copy the old result to
>> -<whatever-we-call-the-unexpected-pass-result>.txt, and check in the new
>> result as -<whatever-we-call-the-expected-failure-result.txt>.
>
>
> I think we should do this. I don't care much about the naming.
>
>>
>> This replaces all other approaches to marking expected failures, including
>> the Skipped list, overwriting -expected even you know the result is a
>> regression, marking the test in TestExpectations as Skip, Wontfix, Image,
>> Text, or Text+Image, or any of the other legacy techniques for marking an
>> expected failure reult.
>
>
> Don't forget suffixing the test with "-disabled"! We have 109 such tests at
> the moment according to
> http://code.google.com/searchframe#search/&exact_package=chromium&q=file:third_party/WebKit/LayoutTests/.*%5C-disabled$&type=cs.
> I think we should also get rid of this. If we need a way to disable a test
> across ports (e.g. because it crashes in cross-platform code), we should
> make a Skipped/TestExpectations file in LayoutTest/platform instead of
> renaming the test file.
>
>
> I agree that renaming to -disabled should be phased out as well. I
> specifically did not cover failure modes that produce no result, such as
> crashes or hangs. Those should still be tracked via TestExpectations IMO.
> Likewise for nondeterministic expectations failures.
>

Since we support cascading TestExpectations now, we can replace
-disabled with a common TestExpectations file that all ports share, as
appropriate (this would require ORWT to recognize TestExpectations, of
course, which I plan to make happen soonish).

-- Dirk