[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Dirk Pranke]

On Fri, Jul 1, 2011 at 2:38 PM, Darin Adler <darin at apple.com> wrote:
> On Jul 1, 2011, at 1:54 PM, Dirk Pranke wrote:
>
>> I do not believe that -expected should be used to track "incorrect" results
>
> A huge percentage of our expected results are incorrect in one sense or another. It’s a massive project to rename them all to reflect the fact that they reflect some amount of failure rather than total success. The project may be worthwhile.
>

Really? Fascinating. Does that apply to -expected.txt files in the
base directories, or just platform-specific exceptions?

I wonder how it is that I've been working (admittedly, mostly on
tooling) in WebKit for more that two years and this is the first I'm
hearing about this. It seems like we're making our jobs much harder by
doing this. Are there reasons we doing things this way apart from not
having a solution along the lines of something like we've been talking
about in this thread?

E.g., a test's results on given port may differ from what might be
correct, but the port may not want to fix this (chromium often uses
WONTFIX in the test_expectations.txt for this) seems like a sensible
reason, but, assuming you run the test at all, I would've prefer that
to be tracked by a -failure.txt sort of solution.

-- Dirk