[webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling "failing" layout tests and TestExpectations

[Ojan Vafai]

On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak <mjs at apple.com> wrote:

> Here's how I imagine the workflow when a sheriff or just innocent
> bystander notices a deterministically failing test. Follow this two-step
> algorithm:
>
> 1) Are you confident that the new result is an improvement or no worse? If
> so, then simply update -expected.txt.
> 2) Otherwise, copy the old result to
> -<whatever-we-call-the-unexpected-pass-result>.txt, and check in the new
> result as -<whatever-we-call-the-expected-failure-result.txt>.
>

I think we should do this. I don't care much about the naming.


> This replaces all other approaches to marking expected failures, including
> the Skipped list, overwriting -expected even you know the result is a
> regression, marking the test in TestExpectations as Skip, Wontfix, Image,
> Text, or Text+Image, or any of the other legacy techniques for marking an
> expected failure reult.
>

Don't forget suffixing the test with "-disabled"! We have 109 such tests at
the moment according to
http://code.google.com/searchframe#search/&exact_package=chromium&q=file:third_party/WebKit/LayoutTests/.*%5C-disabled$&type=cs.
I think we should also get rid of this. If we need a way to disable a test
across ports (e.g. because it crashes in cross-platform code), we should
make a Skipped/TestExpectations file in LayoutTest/platform instead of
renaming the test file.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120821/aad70559/attachment.html>