[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Maciej Stachowiak]

On Jul 5, 2011, at 12:29 PM, Dirk Pranke wrote:

> 
> The problem with your idea is I think what brought this idea up in the
> first place: if you just track that the test is failing using the
> test_expectations.txt file, but don't track *how* it is failing (by
> using something like the -failing.txt idea, or a new -expected.txt
> file), then you cannot tell when the failing output changes, and you
> may miss significant regressions.

That's right, layout tests were designed to be regression tests rather than correctness tests. They are supposed to detect changes in behavior. Having an existing bug is not necessarily a good reason to drop test coverage.

I think instead of introducing a new -failing.txt concept, a better approach would be to have a way to mark in test_expectations.txt that the checked-in -expected.txt for that particular platform represents a bug. I think that is a better way to indicate the state, all in a centralized place, than using a different filename.

Regards,
Maciej


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110705/ebb39fec/attachment.html>