[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Dirk Pranke]

No. Leave the -expected test alone with the "correct but different on
this platform" expectation (of course, if the platform matches the
generic version or some upstream version, you don't need an
-expected.txt file at all).

-- Dirk

On Fri, Jul 1, 2011 at 2:21 PM, Eric Seidel <eric at webkit.org> wrote:
> So then you'd never want both in the same directory, doing so shoudl
> be an error?
>
> -eric
>
> On Fri, Jul 1, 2011 at 2:18 PM, Ojan Vafai <ojan at chromium.org> wrote:
>> What Dirk said. It's just adding another layer into the fallback order.
>>
>> On Fri, Jul 1, 2011 at 1:54 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>>
>>> -failing should trump -expected.
>>>
>>> I also like Ojan's idea.
>>>
>>> I do not believe that -expected should be used to track "incorrect"
>>> results, because that makes understanding how tests are supposed to
>>> run dependent on the knowledge of the bug database as well.
>>>
>>> I think Ryosuke's concern is legitimate, both out of concern for
>>> Chromium's long list of failures and for what would happen if other
>>> ports started also running pixel tests, but I don't know if it's a big
>>> enough concern to kill the idea. It would be interesting to see how
>>> big of an impact there is, and, obviously, a given port could chose
>>> not to use -failure files if it didn't want to.
>>>
>>> -- Dirk
>>>
>>> On Fri, Jul 1, 2011 at 12:56 PM, Eric Seidel <eric at webkit.org> wrote:
>>> > I like the idea of -failing.  But what happens when you have both
>>> > -failing and -expected in the same directory?  Are either accepted?
>>> > (in which case it's like a file-system version of test-expetations
>>> > flaky lists)
>>> >
>>> > On Fri, Jul 1, 2011 at 12:49 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>> >> I proposed a while back to chromium folk that we minimize the use of
>>> >> TEXT
>>> >> and IMAGE and instead check in the failing results the way we do with
>>> >> the
>>> >> non-chromium ports*. I don't like that we rely on bugs to track that
>>> >> the
>>> >> result is incorrect though, so my suggestion was that we change the
>>> >> filename
>>> >> to indicate it. So, foo/bar-expected.txt becomes foo/bar-failing.txt
>>> >> and we
>>> >> just add the -failing version to the fallback order.
>>> >> The main thing I like about this approach is that it allows you to
>>> >> still
>>> >> have a clear list of failing tests that need fixing. I believe that
>>> >> with the
>>> >> current model of checking in a failing result and filing a bug, failing
>>> >> tests are forgotten about.
>>> >> Ojan
>>> >> * My original proposal to Chromium folk wanted to get rid of TEXT and
>>> >> IMAGE
>>> >> entirely from the expectations format. It was generally well received
>>> >> except
>>> >> it it makes handling certain temporary failures considerably more
>>> >> difficult
>>> >> (e.g. pulling in a new version of Skia).
>>> >>
>>> >> On Fri, Jul 1, 2011 at 11:09 AM, Adam Barth <abarth at webkit.org> wrote:
>>> >>>
>>> >>> You can do the same thing with NRWT that you can do with ORWT in this
>>> >>> regard, but nothing new.  The test_expectation.txt file does give you
>>> >>> more fine-grained control than Skipped in the sense that you can
>>> >>> distinguish between TEXT, IMAGE, CRASH, and TIMEOUT failures, but it
>>> >>> doesn't let you distinguish between different sorts of TEXT failures,
>>> >>> for example.
>>> >>>
>>> >>> My sense is that the test_expectation.txt file is already somewhat
>>> >>> over complicated for the problem it solves.  In this case, the
>>> >>> workflow of changing the expected results and filing a bug to track
>>> >>> the failure seems like a reasonable solution, especially if there's a
>>> >>> keyword or master bug that lets you find all these bugs easily.
>>> >>>
>>> >>> Adam
>>> >>>
>>> >>>
>>> >>> On Fri, Jul 1, 2011 at 10:58 AM, Adam Roben <aroben at apple.com> wrote:
>>> >>> > When a test starts failing on a bot that uses old-run-webkit-tests,
>>> >>> > we
>>> >>> > typically check in expected failure results for that test (e.g.,
>>> >>> > <http://trac.webkit.org/changeset/90235>). That way we can find out
>>> >>> > if the
>>> >>> > test's behavior changes in the future even though the test is still
>>> >>> > failing.
>>> >>> > This is particularly useful for tests that are actually testing
>>> >>> > multiple
>>> >>> > things at once. (Maybe they should be broken up into multiple tests,
>>> >>> > but
>>> >>> > that's a different discussion.)
>>> >>> >
>>> >>> > Is there a way to achieve this with new-run-webkit-tests? I know
>>> >>> > that
>>> >>> > you can mark a test as an expected failure (either a text diff, or
>>> >>> > an image
>>> >>> > diff, or both). Does it let you specify that the test should fail in
>>> >>> > a
>>> >>> > particular way?
>>> >>> _______________________________________________
>>> >>> webkit-dev mailing list
>>> >>> webkit-dev at lists.webkit.org
>>> >>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> webkit-dev mailing list
>>> >> webkit-dev at lists.webkit.org
>>> >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>> >>
>>> >>
>>> > _______________________________________________
>>> > webkit-dev mailing list
>>> > webkit-dev at lists.webkit.org
>>> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>> >
>>
>>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>