[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Dirk Pranke]

-failing should trump -expected.

I also like Ojan's idea.

I do not believe that -expected should be used to track "incorrect"
results, because that makes understanding how tests are supposed to
run dependent on the knowledge of the bug database as well.

I think Ryosuke's concern is legitimate, both out of concern for
Chromium's long list of failures and for what would happen if other
ports started also running pixel tests, but I don't know if it's a big
enough concern to kill the idea. It would be interesting to see how
big of an impact there is, and, obviously, a given port could chose
not to use -failure files if it didn't want to.

-- Dirk

On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <eric at webkit.org> wrote:
> I like the idea of -failing.  But what happens when you have both
> -failing and -expected in the same directory?  Are either accepted?
> (in which case it's like a file-system version of test-expetations
> flaky lists)
>
> On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <ojan at chromium.org> wrote:
>> I proposed a while back to chromium folk that we minimize the use of TEXT
>> and IMAGE and instead check in the failing results the way we do with the
>> non-chromium ports*. I don't like that we rely on bugs to track that the
>> result is incorrect though, so my suggestion was that we change the filename
>> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt and we
>> just add the -failing version to the fallback order.
>> The main thing I like about this approach is that it allows you to still
>> have a clear list of failing tests that need fixing. I believe that with the
>> current model of checking in a failing result and filing a bug, failing
>> tests are forgotten about.
>> Ojan
>> * My original proposal to Chromium folk wanted to get rid of TEXT and IMAGE
>> entirely from the expectations format. It was generally well received except
>> it it makes handling certain temporary failures considerably more difficult
>> (e.g. pulling in a new version of Skia).
>>
>> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <abarth at webkit.org> wrote:
>>>
>>> You can do the same thing with NRWT that you can do with ORWT in this
>>> regard, but nothing new.  The test_expectation.txt file does give you
>>> more fine-grained control than Skipped in the sense that you can
>>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it
>>> doesn't let you distinguish between different sorts of TEXT failures,
>>> for example.
>>>
>>> My sense is that the test_expectation.txt file is already somewhat
>>> over complicated for the problem it solves.  In this case, the
>>> workflow of changing the expected results and filing a bug to track
>>> the failure seems like a reasonable solution, especially if there's a
>>> keyword or master bug that lets you find all these bugs easily.
>>>
>>> Adam
>>>
>>>
>>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aroben at apple.com> wrote:
>>> > When a test starts failing on a bot that uses old-run-webkit-tests, we
>>> > typically check in expected failure results for that test (e.g.,
>>> > <http://trac.webkit.org/changeset/90235>). That way we can find out if the
>>> > test's behavior changes in the future even though the test is still failing.
>>> > This is particularly useful for tests that are actually testing multiple
>>> > things at once. (Maybe they should be broken up into multiple tests, but
>>> > that's a different discussion.)
>>> >
>>> > Is there a way to achieve this with new-run-webkit-tests? I know that
>>> > you can mark a test as an expected failure (either a text diff, or an image
>>> > diff, or both). Does it let you specify that the test should fail in a
>>> > particular way?
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>