[webkit-dev] A post-mordem of today's tree redness

[Adam Barth]

On Tue, Apr 6, 2010 at 10:43 AM, Alexey Proskuryakov <ap at webkit.org> wrote:
> On 06.04.2010, at 10:06, Adam Barth wrote:
>>> 4) It's harder to isolate regressions if these appear and disappear
>>> several times (aforementioned confusion doesn't help either). Screening bugs
>>> about regressions also becomes more error-prone. This arguments goes both
>>> ways though - it's even harder to isolate regressions if the platform in
>>> question had broken build at the time.
>>
>> Concretely, supposed we hadn't cleaned up the Tiger bot to be green
>> recently.  I strongly suspect the regression caused by r57081 would
>> have been lost in the thought process that "the Tiger bot is always
>> red."  Even though the regression was real and affected every
>> platform.  Had we noticed the problem (say) a month later, we would
>> have had a devil of a time tracking down the issue as evidenced by the
>> effort required to fix the previous ancient Tiger-only failures.
>
> I was talking about regressions not covered by regression tests. When there
> is a bug about something that works in Safari 4, but not in ToT, you often
> need to find out when exactly it broke - and if there were several breakage
> points, it gets harder. Usually not by much, but sometimes this can lead you
> astray, and then it costs a lot.
>
> Again, this is not to say that we should never roll out patches, but it's
> another cost to consider.

It seems like this consideration pushes us towards rolling out a patch
as quickly as possible if we're going to roll it out at all.  If the
patch is only in the tree for a handful of revisions, that makes is
less likely you'll hit those revisions in your binary search between
Safari 4 and TOT.  Yesterday's events are somewhat of a worse case
scenario for this concern because we rolled out a patch that had been
in the tree for six hours.

Adam