[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Dirk Pranke]

On Tue, Jul 5, 2011 at 1:53 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Jul 5, 2011 1:26 PM, "Dirk Pranke" <dpranke at chromium.org> wrote:
>> > However, we can do the same with the existing testing framework since we
>> > can
>> > associate a test with a bug by adding a line like this:
>> > BUGWK????? my-test.html = PASS
>>
>> You lost me here ...
>
> What I meant is that we can use test_expectations.txt to annotate a test
> with a bug number even if we overrode -expected.txt by adding that line.
> This allows us to track of tests with bad expected results while also
> allowing us to catch any changes in actual result.
>

Ah. That makes a lot of sense. Good idea.

On Tue, Jul 5, 2011 at 3:46 PM, Ojan Vafai <ojan at chromium.org> wrote:
> We could simplify the syntax somewhat to not require the "= PASS" at the
> end. We could also change the bug format to be actual links instead (e.g.
> webkit.org/b/12345 and crbug.com/12345).
> webkit.org/b/12345 : fast/canvas/canvastest.html
> You can also list multiple bug per line:
> webkit.org/b/12345 webkit.org/b/33333 : fast/canvas/canvasttest.html
> That seem OK?

I'm not so much a fan of this change. It's more typing and I'm not
sure if it makes anything much easier for the user (maybe viewing in a
text editor that will automatically hyperlink the text, I guess).

Note that the current syntax does support multiple bugs per line
(although not the hyperlink syntax you propose); I'm sure you knew
that, so perhaps you were just mentioning that for the others on the
list.