[webkit-dev] The tree is on fire: a tragedy of the commons

[Geoffrey Garen]

I think it would be more productive to start with better systems for informing people that they've broken something, and move on to rolling out patches aggressively if informing people doesn't work.

It's not surprising that people neglect a red tree when they don't know about it.

A lot of the proposals on this thread would interfere with this work flow:

1. Finish patch and get it working on local machine.
2. Check in, automatically test for compatibility on other machines and OS's in parallel, resolving unexpected problems as they arise.

and change it to this work flow:

0. Purchase and set up about 15 different build environments.
1. Finish patch and get it working on local machine.
2. Manually test on build environments purchased and set up in (0).
3. Check in.

That would be a serious blow to productivity -- probably a cure that is worse than the disease.

Bear in mind that the build environments problem is multiplied by Google's choice to use a separate JavaScript engine, which effectively almost doubles the testing surface area.

Geoff

On Feb 26, 2010, at 9:44 AM, Eric Seidel wrote:

> The bots take 15 minutes to cycle.  The moment the build is broken,
> thats FIX_TIME + BOT_CYCLE_TIME until their green again.
> 
> I think we should cap the fix grace period at something like 15
> minutes, that means no more than 30 minutes of tree redness per break.
> That might be too aggressive to start with for WebKit, but I think we
> should move towards that.
> 
> I would re-write rule one as something like this:
> 1.  Comment in the bugzilla bug when the build breaks.  If there is no
> bugzilla bug, comment in #webkit.
> 2.  15 minutes after the break or 10 minutes after the comment, with
> no reply from the breaker, roll out the patch.
> 
> -eric
> 
> On Fri, Feb 26, 2010 at 9:32 AM, Nikolas Zimmermann
> <zimmermann at physik.rwth-aachen.de> wrote:
>> 
>> Am 26.02.2010 um 18:17 schrieb Dimitri Glazkov:
>> 
>>> To summarize the thread:
>>> 
>>> 1) We're adopting "when in doubt, roll it out" approach to patches
>>> that turn tree red.
>>> 2) Need to find a way to run Mac-EWS for non-committers.
>>> 3) Enable "build-break" emails to webkit-dev or another opt-in mailing
>>> list
>>> 
>>> What else?
>> 
>> I'm a bit scared of rule 1. How about we define a minimum delay when to
>> roll-out patches, after they break something?
>> Let's say, if a commit breaks the tree, give the commiter a time frame of 30
>> minutes to fix it - otherwhise roll-out (we could even automate that.)
>> 
>> Example: When landing a SVG patch, that worked fine on Leopard, but broke
>> Snow Leopard, I'd like to have some time to identify wheter it's the
>> fault of my patch, or a platform specific problem. If it's the fault of my
>> patch, I have no problem with reverting. But if I can't immediately fix the
>> problem, because it's a platform specific issue, which can not be fixed (in
>> terms of WebKit), then adding to the Skipped list, and filing a new bug
>> just takes 5 minutes. Reverting the whole patch, just to reland it with a
>> Skipped list addition is a bit too much work for me.
>> 
>> What do others think?
>> 
>> Cheers,
>> Niko
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev