[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Adam Barth]

You can do the same thing with NRWT that you can do with ORWT in this
regard, but nothing new.  The test_expectation.txt file does give you
more fine-grained control than Skipped in the sense that you can
distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it
doesn't let you distinguish between different sorts of TEXT failures,
for example.

My sense is that the test_expectation.txt file is already somewhat
over complicated for the problem it solves.  In this case, the
workflow of changing the expected results and filing a bug to track
the failure seems like a reasonable solution, especially if there's a
keyword or master bug that lets you find all these bugs easily.

Adam


On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aroben at apple.com> wrote:
> When a test starts failing on a bot that uses old-run-webkit-tests, we typically check in expected failure results for that test (e.g., <http://trac.webkit.org/changeset/90235>). That way we can find out if the test's behavior changes in the future even though the test is still failing. This is particularly useful for tests that are actually testing multiple things at once. (Maybe they should be broken up into multiple tests, but that's a different discussion.)
>
> Is there a way to achieve this with new-run-webkit-tests? I know that you can mark a test as an expected failure (either a text diff, or an image diff, or both). Does it let you specify that the test should fail in a particular way?