[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Dirk Pranke]

On Fri, Jul 1, 2011 at 3:04 PM, Darin Adler <darin at apple.com> wrote:
> On Jul 1, 2011, at 2:54 PM, Dirk Pranke wrote:
>
>> Does that apply to -expected.txt files in the base directories, or just platform-specific exceptions?
>
> Base directories.
>
> Expected files contain output reflecting the behavior of WebKit at the time the test was checked in. The expected result when we re-run a test. Many expected files contain text that says “FAIL” in them. The fact that these expected results are not successes, but rather expected failures does not seem to me to be a subtle point, but one of the basic things about how these tests are set up.
>
>> I wonder how it is that I've been working (admittedly, mostly on tooling) in WebKit for more that two years and this is the first I'm hearing about this.
>
> I’m guessing it’s because you have been working on Chrome.
>
> The Chrome project came up with a different system for testing layered on top of the original layout test machinery based on different concepts. I don’t think anyone ever discussed that system with me; I was the one who created the original layout test system, to help Dave Hyatt originally, and then later the rest of the team started using it.
>
>> Are there reasons we [are] doing things this way[?]
>
> Sure. The idea of the layout test framework is to check if the code is still behaving as it did when the test was created and last run; we want to detect any changes in behavior that are not expected. When there are expected changes in behavior, we change the contents of the expected results files.

I see, you're using "expected" in the sense of "this is what we expect
to get", not "this is what is correct".

>
> It seems possibly helpful to augment the test system with editorial comments about which tests show bugs that we’d want to fix. But I wouldn’t want to stop running all regression tests where the output reflects the effects of a bug or missing feature.
>

Of course, looking for changes in behavior is entirely the point of
regression testing. Thank you for the history lesson :).

-- Dirk