<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Aug 21, 2012, at 3:23 PM, Ojan Vafai <<a href="mailto:ojan@chromium.org">ojan@chromium.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak <span dir="ltr"><<a href="mailto:mjs@apple.com" target="_blank" class="cremed">mjs@apple.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Here's how I imagine the workflow when a sheriff or just innocent bystander notices a deterministically failing test. Follow this two-step algorithm:<br>
<br>
1) Are you confident that the new result is an improvement or no worse? If so, then simply update -expected.txt.<br>
2) Otherwise, copy the old result to -<whatever-we-call-the-unexpected-pass-result>.txt, and check in the new result as -<whatever-we-call-the-expected-failure-result.txt>.<br></blockquote><div><br></div><div>
I think we should do this. I don't care much about the naming.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This replaces all other approaches to marking expected failures, including the Skipped list, overwriting -expected even you know the result is a regression, marking the test in TestExpectations as Skip, Wontfix, Image, Text, or Text+Image, or any of the other legacy techniques for marking an expected failure reult.<br>
</blockquote><div><br></div><div>Don't forget suffixing the test with "-disabled"! We have 109 such tests at the moment according to <a href="http://code.google.com/searchframe#search/&exact_package=chromium&q=file:third_party/WebKit/LayoutTests/.*%5C-disabled$&type=cs" class="cremed">http://code.google.com/searchframe#search/&exact_package=chromium&q=file:third_party/WebKit/LayoutTests/.*%5C-disabled$&type=cs</a>. I think we should also get rid of this. If we need a way to disable a test across ports (e.g. because it crashes in cross-platform code), we should make a Skipped/TestExpectations file in LayoutTest/platform instead of renaming the test file.</div>
</div></div>
</blockquote></div><br><div>I agree that renaming to -disabled should be phased out as well. I specifically did not cover failure modes that produce no result, such as crashes or hangs. Those should still be tracked via TestExpectations IMO. Likewise for nondeterministic expectations failures.</div><div><br></div><div>Regards,</div><div>Maciej</div><div><br></div></body></html>