[webkit-dev] The tree is on fire: a tragedy of the commons

[Jeremy Orlow]

On Fri, Feb 26, 2010 at 9:00 PM, Jeremy Orlow <jorlow at chromium.org> wrote:

> On Fri, Feb 26, 2010 at 8:53 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>>
>> On Feb 26, 2010, at 11:43 AM, Simon Fraser wrote:
>>
>>
>>> Mozilla has (or at least had when I worked there) two additional "tree
>>> rules" that helped keep the tree green:
>>>
>>> 1. A sheriff was appointed at all times, and had the authority to close
>>> the tree if there was significant build or test breakage. Closing the tree
>>> meant that it was blocked to new commits other than those intended to fix
>>> problems. Closing the tree also sends a strong message that "something is
>>> broken, please pitch in and fix it if you can".
>>>
>>> Sheriff duties were shared around between responsible committers, so as
>>> not to overly burden one person.
>>>
>>
>> I think the build sheriff idea is a good one. Maybe what we want is to
>> have a sheriff responsible for each build train that has an active buildbot.
>> (It could be the same person responsible for several build trains, the main
>> qualification would be having reasonable familiarity with a port and access
>> to its build environment.)
>>
>> However, I am not so sure "close the tree" is necessarily the best focus
>> for sheriff actions. What I'd prefer to see is that the sheriff the person
>> primarily responsible for reverting broken patches if not fixed in a timely
>> manner. Then we could have some human judgment in the process and specific
>> people with clear responsibility.
>
>
> I agree "close to the tree" is not necessary for the reasons you listed.
>  And I think most people from the Chromium would welcome this change
> (sheriff + ability to close).  We've been advocating it for some time now.
>  :-)
>

Oops....I completely misread what you said.

The reason why being able to close the tree is important is because
sometimes it can take a while to sort out what caused what failures.  And
it's important not to allow more breakage in the mean time.  In Chromium, we
often have a good deal of redness, but as long as the sheriffs feel as
though they're on top of it, the tree stays open.  Now, I'll admit that we
have many more long running bots (like memory leak bots) and so these kinds
of train wrecks that require sorting happen way less in WebKit, but it still
might be nice to have the ability when necessary.

The suggestion below (2) about notes on the waterfall sounds great, but we
do OK by abusing the "tree is closed/open" string to keep track of other
state (like who's working on what fix).  We've found this works "good
enough".  And maybe some informal banner like this would be good enough for
the first rev, unless we thought per CL annotations would be easy to
implement.

I'll note that in the Chromium project, we've had a very strong "keep the
tree green" ethic for some time now.  And we have a good deal of experience
related to it.  Certainly there are multiple ways to solve various problems,
but it might be worth taking a look at how we do things to see if there are
other parts of how we do things that might be of interest.


>  2. The Mozilla tinderbox page (their buildbot waterfall) had a way for
>>> people to leave comments, by adding a "star" to a particular build with a
>>> comment. This is used as a way to communicate that someone has noticed the
>>> breakage, and is working on it.
>>>
>>
>> Sounds like a good idea. Wondering if that fits better in the console view
>> or the extensions view.
>>
>>
>>
>>> In general, I think the waterfall page could be improved in order to make
>>> "breakage archeology" easier. Entries in the Changes column should be direct
>>> links to trac changesets, for example.
>>>
>>
>> That sounds good too. Another thing that would help is adding "next page"
>> links to the console view, like we have on the waterfall. The console link
>> often makes it easier to quickly identify the patch that went bad, but only
>> if the badness is recent enough to show up.
>>
>> Regards,
>> Maciej
>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100226/bd513606/attachment.html>