[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?
rniwa at webkit.org
Tue Jul 5 13:21:47 PDT 2011
On Tue, Jul 5, 2011 at 12:29 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> The problem with your idea is I think what brought this idea up in the
> first place: if you just track that the test is failing using the
> test_expectations.txt file, but don't track *how* it is failing (by
> using something like the -failing.txt idea, or a new -expected.txt
> file), then you cannot tell when the failing output changes, and you
> may miss significant regressions.
Even -failing.txt/png won't solve this problem completely if there are
multiple regressions. Consider the following senario:
1. Port has a bug R1 so checks in my-test-failing.png for my-test.html
2. New regression R2 is introduced; and new my-test-failing.png is
3. R2 is fixed
Now what? The only way to know whether R2 was really fixed is by comparing
the current result with the result checked in step 1 by by checking out that
the png committed in step 2.
However, we can do the same with the existing testing framework since we can
associate a test with a bug by adding a line like this:
BUGWK????? my-test.html = PASS
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the webkit-dev