[webkit-dev] Best practices for landing new/changed layout test expectations?

[Ryosuke Niwa]

On Tue, Feb 26, 2013 at 1:55 AM, Tom Hudson <tomhudson at google.com> wrote:

> On Mon, Feb 25, 2013 at 10:34 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>> It should be fairly straight forward to create a tool that analyzes files
>> changed in each commit and deduce which tests' expected results have been
>> changed. The tool can then fetch results from each port' bot for those
>> tests and automatically land them. It can then comment on the bug
>> automatically about these rebaseline commits. There is no need to add &
>> remove entries from TestExpectation files.
>>
>
> I think it's implied in other messages in the thread, but just to be
> explicit: this automated rebaseline commit feels like exactly the wrong
> thing to do. How can you rebaseline a test without SOME manual inspection?
> I know that mass layout rebaselines may not have every pixel checked, but
> there's tooling to help with that.
>
> Changes that are benign in one port may not be benign in other ports.
> Automatically landing changes to other ports' baselines risks corrupting
> our test data.
>

I hate to repeat myself every time this conversation comes up but here it
is: historically, only Chromium port used expected results as "correct
results" and non-Chromium ports used expected results files to store "the
current state of the world" including expected failures although this may
have changed a little since non-Chromium ports started using
TestExpectations instead of Skipped files.

There is a significant danger in adding test expectations as opposed to
committing expected failures because tests with Failure or ImageOnlyFailure
expectation can regress further in any minute. I've encountered several
severe regressions that would have caught if relevant tests had not been
marked Failure or ImageOnlyFailure (in some cases needing rebaselines) for
some minor rendering issues in the past.

So no, it's not exactly the wrong thing to do.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130226/6a69cc76/attachment.html>