[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Dirk Pranke]

On Tue, Jul 5, 2011 at 1:21 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Tue, Jul 5, 2011 at 12:29 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> The problem with your idea is I think what brought this idea up in the
>> first place: if you just track that the test is failing using the
>> test_expectations.txt file, but don't track *how* it is failing (by
>> using something like the -failing.txt idea, or a new -expected.txt
>> file), then you cannot tell when the failing output changes, and you
>> may miss significant regressions.
>
> Even -failing.txt/png won't solve this problem completely if there are
> multiple regressions.  Consider the following senario:
>
> Port has a bug R1 so checks in my-test-failing.png for my-test.html
> New regression R2 is introduced; and new my-test-failing.png is checked in.
> R2 is fixed
>
> Now what?  The only way to know whether R2 was really fixed is by comparing
> the current result with the result checked in step 1 by by checking out that
> the png committed in step 2.

This is true ... you head down the path of tracking a graph of "fails
like X with bug Y still brokenness" and it's not clear where the end
of that path is.

> However, we can do the same with the existing testing framework since we can
> associate a test with a bug by adding a line like this:
> BUGWK????? my-test.html = PASS

You lost me here ...

-- Dirk