[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

[Peter Kasting]

On Wed, Jun 6, 2012 at 11:28 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> > I don't think DIFF is any better.  It sounds like it means the output is
> "different than what we wanted", thus it effectively means "didn't pass",
> and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would
> expect FAIL to.
>
> The output being different implies that we have an output, which isn't
> true when DRT/WTR crashes or times out.
>
Those are outputs as well.  Or if you don't like the work "output"
substitute "outcome".

> > Personally I'd prefer to resolve this -- if we need to -- by removing
> FAIL entirely.  Being explicit about your expectations isn't a bad thing.
>  Plus, the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems
> likely to be small.
>
> People use FAIL when they don't know what to expect; e.g. adding or
> rebaselining tests. zit's utterky unreasonable to expect patch authors to
> add TEXT IMAGE TEXT+IMAGE to every test they're expecting to rebaseline.
>
If you're rebaselining an existing test, presumably the test already has an
expectation that matches what it's doing, or you can see what it's doing to
write one.

If the rebaselining tools can already figure out the right behavior when an
expectation is FAIL, why don't we just make them work correctly regardless
of what the expectation says?  Or work correctly when the expectation is
the empty string?  Those would let people just write "foo.html =" and
rebaseline which is even easier on them.

> I also think it's a bad practice to add test expections just to keep bots
> green. It's much better if the tests started to fail on the waterfall so
> that people who pay attention to bots can rebaseline them since most people
> forget about rebaselining tests once their patches are landed.
>

I can't tell what you're arguing here.  As far as I know nothing in what I
was saying related to the question of the reason why people are adding
expectations.

PK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120607/fa114c8d/attachment.html>