[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

[Ryosuke Niwa]

On Thu, Jun 7, 2012 at 11:03 AM, Peter Kasting <pkasting at chromium.org>wrote:

> On Wed, Jun 6, 2012 at 11:28 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>> > I don't think DIFF is any better.  It sounds like it means the output
>> is "different than what we wanted", thus it effectively means "didn't
>> pass", and one would expect it to match MISSING/CRASH/TIMEOUT as much as
>> one would expect FAIL to.
>>
>> The output being different implies that we have an output, which isn't
>> true when DRT/WTR crashes or times out.
>>
> Those are outputs as well.  Or if you don't like the work "output"
> substitute "outcome".
>

I don't buy it.

>  > Personally I'd prefer to resolve this -- if we need to -- by removing
>> FAIL entirely.  Being explicit about your expectations isn't a bad thing.
>>  Plus, the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems
>> likely to be small.
>>
>> People use FAIL when they don't know what to expect; e.g. adding or
>> rebaselining tests. zit's utterky unreasonable to expect patch authors to
>> add TEXT IMAGE TEXT+IMAGE to every test they're expecting to rebaseline.
>>
> If you're rebaselining an existing test, presumably the test already has
> an expectation that matches what it's doing, or you can see what it's doing
> to write one.
>

Not if the test was padding. I'm talking about the case where you're
modifying WebCore and know that some tests are going to need rebaselines.
People have advised in the past that patch authors add failing test
expectations to TestExpectations files to avoid turning bots red.

If the rebaselining tools can already figure out the right behavior when an
> expectation is FAIL, why don't we just make them work correctly regardless
> of what the expectation says?  Or work correctly when the expectation is
> the empty string?  Those would let people just write "foo.html =" and
> rebaseline which is even easier on them.
>

That isn't really the issue. The problem is that people add test
expectations pre-emptively to keep bots green but end up adding wrong
expectations sometimes because FAIL doesn't encompass MISSING, and that's
the failure you'll get on qt and gtk if you're adding expected results to
mac. If you're adding expected results to just chromium linux, for example,
then you'll get MISSING failure on all but chromium linux.

>  I also think it's a bad practice to add test expections just to keep
>> bots green. It's much better if the tests started to fail on the waterfall
>> so that people who pay attention to bots can rebaseline them since most
>> people forget about rebaselining tests once their patches are landed.
>>
> I can't tell what you're arguing here.  As far as I know nothing in what I
> was saying related to the question of the reason why people are adding
> expectations.
>

It was nothing to do with your response. I was making a general statement.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120607/d7a6b330/attachment.html>