[webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

[Ryosuke Niwa]

On Thu, Jun 7, 2012 at 12:44 PM, Peter Kasting <pkasting at chromium.org>wrote:

> On Thu, Jun 7, 2012 at 12:33 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>> Not if the test was padding. I'm talking about the case where you're
>> modifying WebCore and know that some tests are going to need rebaselines.
>> People have advised in the past that patch authors add failing test
>> expectations to TestExpectations files to avoid turning bots red.
>>
>
> I think this is bad advice.  When I've been sheriff it seems like people
> who try this inevitably miss some tests and platforms anyway, get wrong
> expectations, etc.  All this does is add more work for the submitter that
> is difficult to check ahead of time.  We should just advise people to land
> and then fix (and be around on IRC/notify sheriffs about what's going on).
>

Yes. I'd strongly advocate for not adding test expectations. I've seen too
many patch authors adding test expectations and then forgetting about them.

However, there is a practical problem that the commit queue uses Chromium
Linux port and rejects patches that need rebaselines unless the authors add
test expectations.

It seems like this is what you were saying as your general statement as
> well.
>

Right.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120607/6ce9328c/attachment.html>