[webkit-dev] Best practices for landing new/changed layout test expectations?

[Ryosuke Niwa]

On Mon, Feb 25, 2013 at 6:28 PM, Silvia Pfeiffer <silviapf at chromium.org>wrote:

> On Tue, Feb 26, 2013 at 9:34 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>> It should be fairly straight forward to create a tool that analyzes files
>> changed in each commit and deduce which tests' expected results have been
>> changed. The tool can then fetch results from each port' bot for those
>> tests and automatically land them. It can then comment on the bug
>> automatically about these rebaseline commits. There is no need to add &
>> remove entries from TestExpectation files.
>>
>
> The consequence of this is that it's possible that rebaselines get
> committed that may actually indicate a new bug. Since your suggestion is to
> attach this to the bug through which the patch landed, I think that's ok
> and the original author (or the reviewer) should feel obliged to follow up
> on the newly introduced bug by creating a new bug.
>
> If we can do this in a way that avoids the bots from going red, I think
> it's a good solution.
>

I don't like a dogmatic approach like keeping bots green at all cost.

We need to weigh costs of bots turning red versus asking every contributor
to do extra things in order to keep bots green while noticing that some of
our 400 contributors are bound to make mistakes every now and then.  After
all, those people who get affected by red bots are also people who submit
patches and (hopefully) maintain bots.

I'm also saddened by the fact that the first thing some people do when they
see builds failing or some tests failing is to blame others or complain
instead of trying to fix those problems themselves.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130225/21eeb8ea/attachment.html>