[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Eric Seidel]

So then you'd never want both in the same directory, doing so shoudl
be an error?

-eric

On Fri, Jul 1, 2011 at 2:18 PM, Ojan Vafai <ojan at chromium.org> wrote:
> What Dirk said. It's just adding another layer into the fallback order.
>
> On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> -failing should trump -expected.
>>
>> I also like Ojan's idea.
>>
>> I do not believe that -expected should be used to track "incorrect"
>> results, because that makes understanding how tests are supposed to
>> run dependent on the knowledge of the bug database as well.
>>
>> I think Ryosuke's concern is legitimate, both out of concern for
>> Chromium's long list of failures and for what would happen if other
>> ports started also running pixel tests, but I don't know if it's a big
>> enough concern to kill the idea. It would be interesting to see how
>> big of an impact there is, and, obviously, a given port could chose
>> not to use -failure files if it didn't want to.
>>
>> -- Dirk
>>
>> On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <eric at webkit.org> wrote:
>> > I like the idea of -failing.  But what happens when you have both
>> > -failing and -expected in the same directory?  Are either accepted?
>> > (in which case it's like a file-system version of test-expetations
>> > flaky lists)
>> >
>> > On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <ojan at chromium.org> wrote:
>> >> I proposed a while back to chromium folk that we minimize the use of
>> >> TEXT
>> >> and IMAGE and instead check in the failing results the way we do with
>> >> the
>> >> non-chromium ports*. I don't like that we rely on bugs to track that
>> >> the
>> >> result is incorrect though, so my suggestion was that we change the
>> >> filename
>> >> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt
>> >> and we
>> >> just add the -failing version to the fallback order.
>> >> The main thing I like about this approach is that it allows you to
>> >> still
>> >> have a clear list of failing tests that need fixing. I believe that
>> >> with the
>> >> current model of checking in a failing result and filing a bug, failing
>> >> tests are forgotten about.
>> >> Ojan
>> >> * My original proposal to Chromium folk wanted to get rid of TEXT and
>> >> IMAGE
>> >> entirely from the expectations format. It was generally well received
>> >> except
>> >> it it makes handling certain temporary failures considerably more
>> >> difficult
>> >> (e.g. pulling in a new version of Skia).
>> >>
>> >> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <abarth at webkit.org> wrote:
>> >>>
>> >>> You can do the same thing with NRWT that you can do with ORWT in this
>> >>> regard, but nothing new.  The test_expectation.txt file does give you
>> >>> more fine-grained control than Skipped in the sense that you can
>> >>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it
>> >>> doesn't let you distinguish between different sorts of TEXT failures,
>> >>> for example.
>> >>>
>> >>> My sense is that the test_expectation.txt file is already somewhat
>> >>> over complicated for the problem it solves.  In this case, the
>> >>> workflow of changing the expected results and filing a bug to track
>> >>> the failure seems like a reasonable solution, especially if there's a
>> >>> keyword or master bug that lets you find all these bugs easily.
>> >>>
>> >>> Adam
>> >>>
>> >>>
>> >>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aroben at apple.com> wrote:
>> >>> > When a test starts failing on a bot that uses old-run-webkit-tests,
>> >>> > we
>> >>> > typically check in expected failure results for that test (e.g.,
>> >>> > <http://trac.webkit.org/changeset/90235>). That way we can find out
>> >>> > if the
>> >>> > test's behavior changes in the future even though the test is still
>> >>> > failing.
>> >>> > This is particularly useful for tests that are actually testing
>> >>> > multiple
>> >>> > things at once. (Maybe they should be broken up into multiple tests,
>> >>> > but
>> >>> > that's a different discussion.)
>> >>> >
>> >>> > Is there a way to achieve this with new-run-webkit-tests? I know
>> >>> > that
>> >>> > you can mark a test as an expected failure (either a text diff, or
>> >>> > an image
>> >>> > diff, or both). Does it let you specify that the test should fail in
>> >>> > a
>> >>> > particular way?
>> >>> _______________________________________________
>> >>> webkit-dev mailing list
>> >>> webkit-dev at lists.webkit.org
>> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >>
>> >>
>> >> _______________________________________________
>> >> webkit-dev mailing list
>> >> webkit-dev at lists.webkit.org
>> >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >>
>> >>
>> > _______________________________________________
>> > webkit-dev mailing list
>> > webkit-dev at lists.webkit.org
>> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> >
>
>