[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Eric Seidel]

I like the idea of -failing.  But what happens when you have both
-failing and -expected in the same directory?  Are either accepted?
(in which case it's like a file-system version of test-expetations
flaky lists)

On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <ojan at chromium.org> wrote:
> I proposed a while back to chromium folk that we minimize the use of TEXT
> and IMAGE and instead check in the failing results the way we do with the
> non-chromium ports*. I don't like that we rely on bugs to track that the
> result is incorrect though, so my suggestion was that we change the filename
> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we
> just add the -failing version to the fallback order.
> The main thing I like about this approach is that it allows you to still
> have a clear list of failing tests that need fixing. I believe that with the
> current model of checking in a failing result and filing a bug, failing
> tests are forgotten about.
> Ojan
> * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE
> entirely from the expectations format. It was generally well received except
> it it makes handling certain temporary failures considerably more difficult
> (e.g. pulling in a new version of Skia).
>
> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <abarth at webkit.org> wrote:
>>
>> You can do the same thing with NRWT that you can do with ORWT in this
>> regard, but nothing new.  The test_expectation.txt file does give you
>> more fine-grained control than Skipped in the sense that you can
>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it
>> doesn't let you distinguish between different sorts of TEXT failures,
>> for example.
>>
>> My sense is that the test_expectation.txt file is already somewhat
>> over complicated for the problem it solves.  In this case, the
>> workflow of changing the expected results and filing a bug to track
>> the failure seems like a reasonable solution, especially if there's a
>> keyword or master bug that lets you find all these bugs easily.
>>
>> Adam
>>
>>
>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aroben at apple.com> wrote:
>> > When a test starts failing on a bot that uses old-run-webkit-tests, we
>> > typically check in expected failure results for that test (e.g.,
>> > <http://trac.webkit.org/changeset/90235>). That way we can find out if the
>> > test's behavior changes in the future even though the test is still failing.
>> > This is particularly useful for tests that are actually testing multiple
>> > things at once. (Maybe they should be broken up into multiple tests, but
>> > that's a different discussion.)
>> >
>> > Is there a way to achieve this with new-run-webkit-tests? I know that
>> > you can mark a test as an expected failure (either a text diff, or an image
>> > diff, or both). Does it let you specify that the test should fail in a
>> > particular way?
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>