[webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling "failing" layout tests and TestExpectations
ojan at chromium.org
Tue Aug 21 15:23:20 PDT 2012
On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak <mjs at apple.com> wrote:
> Here's how I imagine the workflow when a sheriff or just innocent
> bystander notices a deterministically failing test. Follow this two-step
> 1) Are you confident that the new result is an improvement or no worse? If
> so, then simply update -expected.txt.
> 2) Otherwise, copy the old result to
> -<whatever-we-call-the-unexpected-pass-result>.txt, and check in the new
> result as -<whatever-we-call-the-expected-failure-result.txt>.
I think we should do this. I don't care much about the naming.
> This replaces all other approaches to marking expected failures, including
> the Skipped list, overwriting -expected even you know the result is a
> regression, marking the test in TestExpectations as Skip, Wontfix, Image,
> Text, or Text+Image, or any of the other legacy techniques for marking an
> expected failure reult.
Don't forget suffixing the test with "-disabled"! We have 109 such tests at
the moment according to
I think we should also get rid of this. If we need a way to disable a test
across ports (e.g. because it crashes in cross-platform code), we should
make a Skipped/TestExpectations file in LayoutTest/platform instead of
renaming the test file.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev