[webkit-dev] Does NRWT let you indicate that a test should fail with a particular failure diff?

[Ryosuke Niwa]

On Tue, Jul 5, 2011 at 3:46 PM, Ojan Vafai <ojan at chromium.org> wrote:
>
> So, you could currently add a line like:
> BUGWK12345 : fast/canvas/canvastest.html = PASS
>
> We could simplify the syntax somewhat to not require the "= PASS" at the
> end.
>

How about renaming PASS to MATCH?  MATCH will tell us that the actual result
matches the expected results but doesn't give us a false impression that the
test is "passing" with a correct output.

We could also change the bug format to be actual links instead (e.g.
> webkit.org/b/12345 and crbug.com/12345).
>
> webkit.org/b/12345 : fast/canvas/canvastest.html
>
> You can also list multiple bug per line:
>
> webkit.org/b/12345 webkit.org/b/33333 : fast/canvas/canvasttest.html
>
> That seem OK?
>

That seems like a huge improvement over the current BUGWK / BUGCR format.

Slightly OT: I'm increasingly convinced that we should try to find a scheme
> that is something in between the Chromium and the non-Chromium ports
> approaches and standardize on a simplified test_expectations format. That's
> certainly a separate discussion, but agreeing on this would be a step in
> that direction.
>

Yes! I'd love to have that discussion on a separate thread.  The difference
between Chromium and non-Chromium ports is a huge pain when rebaselining
tests.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110705/31f73dee/attachment.html>