[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

[Ryosuke Niwa]

On Jun 6, 2012 10:36 PM, "Peter Kasting" <pkasting at chromium.org> wrote:
>
> On Wed, Jun 6, 2012 at 10:22 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>
>> Now that everyone knows the problem, I propose to rename FAIL to DIFF.
>>
>> FAIL should mean that the test fails, not that it fails with image,
text, or image and text failures.
>>
>> DIFF, on the other hand, has no ambiguity. It can't be interpreted as
timeout, crash, or pass but can easily be associated with image and text
differences.
>
>
> I don't think DIFF is any better.  It sounds like it means the output is
"different than what we wanted", thus it effectively means "didn't pass",
and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would
expect FAIL to.

The output being different implies that we have an output, which isn't true
when DRT/WTR crashes or times out.

> Personally I'd prefer to resolve this -- if we need to -- by removing
FAIL entirely.  Being explicit about your expectations isn't a bad thing.
 Plus, the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems
likely to be small.

People use FAIL when they don't know what to expect; e.g. adding or
rebaselining tests. zit's utterky unreasonable to expect patch authors to
add TEXT IMAGE TEXT+IMAGE to every test they're expecting to rebaseline.

I also think it's a bad practice to add test expections just to keep bots
green. It's much better if the tests started to fail on the waterfall so
that people who pay attention to bots can rebaseline them since most people
forget about rebaselining tests once their patches are landed.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120606/643be41d/attachment.html>