[webkit-dev] The tree is on fire: a tragedy of the commons

[Maciej Stachowiak]

On Feb 26, 2010, at 1:36 AM, Adam Barth wrote:

> Not to point fingers, but we've been having trouble keeping
> build.webkit.org green these past few weeks.  As I write this message,
> every platform is broken, again.  As the project scales, polluting the
> build with brokenness impacts more developers and drains more
> productivity.
>
> Here are some approaches we could use to turn this tragedy of the
> commons around:
>
> 1) Adopt a "rollout first, ask questions later" ethic.  The vast
> majority of changes are not important enough to break the build for
> everyone else.  If we adopt a "rollout first, ask questions later"
> ethic, committers would feel free to rollout brokenness to unbreak the
> build and contributors shouldn't be offended if their patch is rolled
> out without their knowledge.  We can always re-land the broken patch
> later once it actually works.
>
> 2) Require pre-commit vetting of patches.  We have the resources to
> build and test every patch on at least one platform before landing the
> patch in the main tree.  Vetting patches before landing will help us
> avoid breaking every platform at once.  Once the patch has been
> vetted, it can either be landed mechanically (i.e., by commit-queue)
> or manually.
>
> Here's how I would design the life and times of a patch:
>
> 1) Contributor uploads patch and nominates it for review.
> 2) Patch vetted by the EWS on numerous platforms.
> 3) If the EWS finds a problem, return to step 1.
> 4) Reviewer marks patch review+.
> 5) Committer decides the patch is ready to land.
> 6) Patch built and tested against top-of-tree on at least one  
> platform.
> 7) If the patch fail to build or pass tests, return to step 1.
> 8) Patch landed.
> 9) If the patch turns any of the "core builders" red, patch is rolled
> out, return to step 1.
>
> I suspect most of our brokenness coming from committers skipping  
> steps 6 and 7.

One data point: I broke the build this weekend, because I introduced a  
problem that affected debug builds but not release. I did a full  
release build on my own system before committing. When someone pointed  
out the breakage, I rolled the patch out myself until I could fix it.  
If the problems were such that I could fix them as quickly as rolling  
out, I would

I feel like the biggest failure in my case was that I forgot to look  
at the bot once my patch went through a cycle. This is why I wish it  
would do some form of more active notification. Sometimes I get  
distracted after committing and forget to keep hitting reload on the  
buildbot page.

Regards,
Maciej