[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Darin Adler]

On Jul 1, 2011, at 3:24 PM, Darin Fisher wrote:

> The granular annotations (more than just SKIP) in test_expectations.txt was something we introduced back when Chrome was failing a large percentage of layout tests, and we needed a system to help us triage the failures.

I do think that a system more sophisticated than our original one is a good idea. Generally speaking, I can only think of one reason to consider not running a test at all:

    1) The test causes a crash or runs excessively slow as part of how it fails, so it makes running tests unwieldy.

And only two reasons to not compare the results of the test with the expected output:

    2) The test fails in unpredictably different ways so has no value as a regression test.
    3) There are no expected results for the test; no one has generated them yet, or there is some reason to not check them in.

None of this has anything to with success or failure, in my opinion.

I think it’s great to be organized about the tests that are run. It would be wonderful to associate bugs with each set of known failures in tests, whether the failures are cross-platform or platform-specific. But I don’t think they way the test is run and compared with expected output should have anything to do with whether it’s an expected success or an expected failure.

    -- Darin