[webkit-dev] The tree is on fire: a tragedy of the commons

[Adam Barth]

Not to point fingers, but we've been having trouble keeping
build.webkit.org green these past few weeks.  As I write this message,
every platform is broken, again.  As the project scales, polluting the
build with brokenness impacts more developers and drains more
productivity.

Here are some approaches we could use to turn this tragedy of the
commons around:

1) Adopt a "rollout first, ask questions later" ethic.  The vast
majority of changes are not important enough to break the build for
everyone else.  If we adopt a "rollout first, ask questions later"
ethic, committers would feel free to rollout brokenness to unbreak the
build and contributors shouldn't be offended if their patch is rolled
out without their knowledge.  We can always re-land the broken patch
later once it actually works.

2) Require pre-commit vetting of patches.  We have the resources to
build and test every patch on at least one platform before landing the
patch in the main tree.  Vetting patches before landing will help us
avoid breaking every platform at once.  Once the patch has been
vetted, it can either be landed mechanically (i.e., by commit-queue)
or manually.

Here's how I would design the life and times of a patch:

1) Contributor uploads patch and nominates it for review.
2) Patch vetted by the EWS on numerous platforms.
3) If the EWS finds a problem, return to step 1.
4) Reviewer marks patch review+.
5) Committer decides the patch is ready to land.
6) Patch built and tested against top-of-tree on at least one platform.
7) If the patch fail to build or pass tests, return to step 1.
8) Patch landed.
9) If the patch turns any of the "core builders" red, patch is rolled
out, return to step 1.

I suspect most of our brokenness coming from committers skipping steps 6 and 7.

Adam