[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Ojan Vafai]

On Tue, Jul 5, 2011 at 3:22 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Jul 5, 2011, at 12:29 PM, Dirk Pranke wrote:
>
>
> The problem with your idea is I think what brought this idea up in the
> first place: if you just track that the test is failing using the
> test_expectations.txt file, but don't track *how* it is failing (by
> using something like the -failing.txt idea, or a new -expected.txt
> file), then you cannot tell when the failing output changes, and you
> may miss significant regressions.
>
>
> That's right, layout tests were designed to be regression tests rather than
> correctness tests. They are supposed to detect changes in behavior. Having
> an existing bug is not necessarily a good reason to drop test coverage.
>
> I think instead of introducing a new -failing.txt concept, a better
> approach would be to have a way to mark in test_expectations.txt that the
> checked-in -expected.txt for that particular platform represents a bug. I
> think that is a better way to indicate the state, all in a centralized
> place, than using a different filename.
>

I was originally opposed to this because I was worried that the listings in
the expectations file would get stale. I still think that's true, but it'll
just be something we need to manage. Changing the filename doesn't ensure
lack of staleness anyways and has the downside that it lacks the extra
explanation you'd find in a bug.

We can do this already with the text_expectations syntax and it will
automatically show the links to the bug on the flakiness dashboard. It would
be straightforward to add a new view to the dashboard that lists all these
tests in an easy to digest form.

So, you could currently add a line like:
BUGWK12345 : fast/canvas/canvastest.html = PASS

We could simplify the syntax somewhat to not require the "= PASS" at the
end. We could also change the bug format to be actual links instead (e.g.
webkit.org/b/12345 and crbug.com/12345).

webkit.org/b/12345 : fast/canvas/canvastest.html

You can also list multiple bug per line:

webkit.org/b/12345 webkit.org/b/33333 : fast/canvas/canvasttest.html

That seem OK?

Slightly OT: I'm increasingly convinced that we should try to find a scheme
that is something in between the Chromium and the non-Chromium ports
approaches and standardize on a simplified test_expectations format. That's
certainly a separate discussion, but agreeing on this would be a step in
that direction.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110705/08f38c9b/attachment.html>