[webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling "failing" layout tests and TestExpectations

[Maciej Stachowiak]

On Aug 21, 2012, at 3:23 PM, Ojan Vafai <ojan at chromium.org> wrote:

> On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> Here's how I imagine the workflow when a sheriff or just innocent bystander notices a deterministically failing test. Follow this two-step algorithm:
> 
> 1) Are you confident that the new result is an improvement or no worse? If so, then simply update -expected.txt.
> 2) Otherwise, copy the old result to -<whatever-we-call-the-unexpected-pass-result>.txt, and check in the new result as -<whatever-we-call-the-expected-failure-result.txt>.
> 
> I think we should do this. I don't care much about the naming.
>  
> This replaces all other approaches to marking expected failures, including the Skipped list, overwriting -expected even you know the result is a regression, marking the test in TestExpectations as Skip, Wontfix, Image, Text, or Text+Image, or any of the other legacy techniques for marking an expected failure reult.
> 
> Don't forget suffixing the test with "-disabled"! We have 109 such tests at the moment according to http://code.google.com/searchframe#search/&exact_package=chromium&q=file:third_party/WebKit/LayoutTests/.*%5C-disabled$&type=cs. I think we should also get rid of this. If we need a way to disable a test across ports (e.g. because it crashes in cross-platform code), we should make a Skipped/TestExpectations file in LayoutTest/platform instead of renaming the test file.

I agree that renaming to -disabled should be phased out as well. I specifically did not cover failure modes that produce no result, such as crashes or hangs. Those should still be tracked via TestExpectations IMO. Likewise for nondeterministic expectations failures.

Regards,
Maciej

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120821/772a65c9/attachment.html>