[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Ryosuke Niwa]

On Tue, Jul 5, 2011 at 12:29 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>
> The problem with your idea is I think what brought this idea up in the
> first place: if you just track that the test is failing using the
> test_expectations.txt file, but don't track *how* it is failing (by
> using something like the -failing.txt idea, or a new -expected.txt
> file), then you cannot tell when the failing output changes, and you
> may miss significant regressions.
>

Even -failing.txt/png won't solve this problem completely if there are
multiple regressions.  Consider the following senario:

   1. Port has a bug R1 so checks in my-test-failing.png for my-test.html
   2. New regression R2 is introduced; and new my-test-failing.png is
   checked in.
   3. R2 is fixed

Now what?  The only way to know whether R2 was really fixed is by comparing
the current result with the result checked in step 1 by by checking out that
the png committed in step 2.

However, we can do the same with the existing testing framework since we can
associate a test with a bug by adding a line like this:
BUGWK????? my-test.html = PASS

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110705/8df004b5/attachment.html>