[webkit-dev] The tree is on fire: a tragedy of the commons

[Simon Fraser]

On Feb 26, 2010, at 11:34 AM, Geoffrey Garen wrote:

>> There is a non-trivial cost of this workflow on the rest of the team.
>> -keeps the commit-queue from running
>> -often results in new test failures going unnoticed because the tree is already red
>> -we can't generally trust that all the tests should pass locally
> 
> I think all of the costs you list fundamentally derive from "failures going unnoticed." That's the rationale for my suggestion that we start by making sure that failures are noticed.
> 
>> Would it be enough for you if you could send a patch to the EWS and get back the results for any test failures?
> 
> It would certainly be very helpful.
> 
> I don't know if it would be enough to make me think a harsh policy of rolling out patches was a good idea.
> 
> But if we had a good system for making failures noticed, and a working EWS, and we still had problems with a red tree, I'm sure I would support some further action to solve the problem.

Mozilla has (or at least had when I worked there) two additional "tree rules" that helped keep the tree green:

1. A sheriff was appointed at all times, and had the authority to close the tree if there was significant build or test breakage. Closing the tree meant that it was blocked to new commits other than those intended to fix problems. Closing the tree also sends a strong message that "something is broken, please pitch in and fix it if you can".

Sheriff duties were shared around between responsible committers, so as not to overly burden one person.

2. The Mozilla tinderbox page (their buildbot waterfall) had a way for people to leave comments, by adding a "star" to a particular build with a comment. This is used as a way to communicate that someone has noticed the breakage, and is working on it.

In general, I think the waterfall page could be improved in order to make "breakage archeology" easier. Entries in the Changes column should be direct links to trac changesets, for example.

Simon