[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Ojan Vafai]

What Dirk said. It's just adding another layer into the fallback order.

On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> -failing should trump -expected.
>
> I also like Ojan's idea.
>
> I do not believe that -expected should be used to track "incorrect"
> results, because that makes understanding how tests are supposed to
> run dependent on the knowledge of the bug database as well.
>
> I think Ryosuke's concern is legitimate, both out of concern for
> Chromium's long list of failures and for what would happen if other
> ports started also running pixel tests, but I don't know if it's a big
> enough concern to kill the idea. It would be interesting to see how
> big of an impact there is, and, obviously, a given port could chose
> not to use -failure files if it didn't want to.
>
> -- Dirk
>
> On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <eric at webkit.org> wrote:
> > I like the idea of -failing.  But what happens when you have both
> > -failing and -expected in the same directory?  Are either accepted?
> > (in which case it's like a file-system version of test-expetations
> > flaky lists)
> >
> > On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <ojan at chromium.org> wrote:
> >> I proposed a while back to chromium folk that we minimize the use of
> TEXT
> >> and IMAGE and instead check in the failing results the way we do with
> the
> >> non-chromium ports*. I don't like that we rely on bugs to track that the
> >> result is incorrect though, so my suggestion was that we change the
> filename
> >> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and
> we
> >> just add the -failing version to the fallback order.
> >> The main thing I like about this approach is that it allows you to still
> >> have a clear list of failing tests that need fixing. I believe that with
> the
> >> current model of checking in a failing result and filing a bug, failing
> >> tests are forgotten about.
> >> Ojan
> >> * My original proposal to Chromium folk wanted to get rid of TEXT and
> IMAGE
> >> entirely from the expectations format. It was generally well received
> except
> >> it it makes handling certain temporary failures considerably more
> difficult
> >> (e.g. pulling in a new version of Skia).
> >>
> >> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <abarth at webkit.org> wrote:
> >>>
> >>> You can do the same thing with NRWT that you can do with ORWT in this
> >>> regard, but nothing new.  The test_expectation.txt file does give you
> >>> more fine-grained control than Skipped in the sense that you can
> >>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it
> >>> doesn't let you distinguish between different sorts of TEXT failures,
> >>> for example.
> >>>
> >>> My sense is that the test_expectation.txt file is already somewhat
> >>> over complicated for the problem it solves.  In this case, the
> >>> workflow of changing the expected results and filing a bug to track
> >>> the failure seems like a reasonable solution, especially if there's a
> >>> keyword or master bug that lets you find all these bugs easily.
> >>>
> >>> Adam
> >>>
> >>>
> >>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aroben at apple.com> wrote:
> >>> > When a test starts failing on a bot that uses old-run-webkit-tests,
> we
> >>> > typically check in expected failure results for that test (e.g.,
> >>> > <http://trac.webkit.org/changeset/90235>). That way we can find out
> if the
> >>> > test's behavior changes in the future even though the test is still
> failing.
> >>> > This is particularly useful for tests that are actually testing
> multiple
> >>> > things at once. (Maybe they should be broken up into multiple tests,
> but
> >>> > that's a different discussion.)
> >>> >
> >>> > Is there a way to achieve this with new-run-webkit-tests? I know that
> >>> > you can mark a test as an expected failure (either a text diff, or an
> image
> >>> > diff, or both). Does it let you specify that the test should fail in
> a
> >>> > particular way?
> >>> _______________________________________________
> >>> webkit-dev mailing list
> >>> webkit-dev at lists.webkit.org
> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >>
> >>
> >> _______________________________________________
> >> webkit-dev mailing list
> >> webkit-dev at lists.webkit.org
> >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >>
> >>
> > _______________________________________________
> > webkit-dev mailing list
> > webkit-dev at lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110701/0d754320/attachment.html>