[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Ojan Vafai]

I proposed a while back to chromium folk that we minimize the use of TEXT
and IMAGE and instead check in the failing results the way we do with the
non-chromium ports*. I don't like that we rely on bugs to track that the
result is incorrect though, so my suggestion was that we change the filename
to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we
just add the -failing version to the fallback order.

The main thing I like about this approach is that it allows you to still
have a clear list of failing tests that need fixing. I believe that with the
current model of checking in a failing result and filing a bug, failing
tests are forgotten about.

Ojan

* My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE
entirely from the expectations format. It was generally well received except
it it makes handling certain temporary failures considerably more difficult
(e.g. pulling in a new version of Skia).

On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <abarth at webkit.org> wrote:

> You can do the same thing with NRWT that you can do with ORWT in this
> regard, but nothing new.  The test_expectation.txt file does give you
> more fine-grained control than Skipped in the sense that you can
> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it
> doesn't let you distinguish between different sorts of TEXT failures,
> for example.
>
> My sense is that the test_expectation.txt file is already somewhat
> over complicated for the problem it solves.  In this case, the
> workflow of changing the expected results and filing a bug to track
> the failure seems like a reasonable solution, especially if there's a
> keyword or master bug that lets you find all these bugs easily.
>
> Adam
>
>
> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aroben at apple.com> wrote:
> > When a test starts failing on a bot that uses old-run-webkit-tests, we
> typically check in expected failure results for that test (e.g., <
> http://trac.webkit.org/changeset/90235>). That way we can find out if the
> test's behavior changes in the future even though the test is still failing.
> This is particularly useful for tests that are actually testing multiple
> things at once. (Maybe they should be broken up into multiple tests, but
> that's a different discussion.)
> >
> > Is there a way to achieve this with new-run-webkit-tests? I know that you
> can mark a test as an expected failure (either a text diff, or an image
> diff, or both). Does it let you specify that the test should fail in a
> particular way?
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110701/de8abe67/attachment.html>