[webkit-dev] The green tree era

[Adam Barth]

On Sat, Apr 3, 2010 at 11:30 PM, Darin Adler <darin at apple.com> wrote:
> On Apr 3, 2010, at 10:36 AM, Adam Barth wrote:
>> Keeping the tree green will require a cultural shift in the project, but I think the near term costs of changing the culture are outweighed by the long term gains in productivity.
>
> Typically a cultural shift would come after some sort of group discussion, rather than being announced on the mailing list as a fait accompli.

We've been discussing this on the mailing list for a while.  Here are
some recent threads:

* https://lists.webkit.org/pipermail/webkit-dev/2009-September/010023.html
In which the conclusion is that we should fix test flakiness rather
than skipping the tests or letting the flakiness persist.

* https://lists.webkit.org/pipermail/webkit-dev/2010-January/011430.html
In which we manually go through the process outlined above using
webkit-dev instead of more targeted channels.

* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011561.html
In which we discuss how best to draw attention to buildbot failures.

* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011562.html
This email in particular, in which Maciej asks someone to build a bot
that does precisely what sheriffbot does:
[[
I'd actually like to see it email a mailing list, in addition to the
individuals it guesses are to blame. That could be either webkit-dev
or a new list. Maybe some won't want the spam but I bet a lot of
people would like to find out about every build break. If it's at all
possible, it would be great to email all of the patch author, the
reviewer and the committer (if different from the patch author).

I also think it would be neat if we could have a bot that alerts about
build breaks on IRC in #webkit.

And finally, it might be good to have extra notice if a build remains
broken for some time (every 24 hours maybe?)
]]

* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011749.html
This long thread where we discuss the motivations for keeping the tree
green and the possibility of using rollouts more than we already are.

Here are some important messages from that thread:
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011765.html
In which Dimitri Glazkov supports a "rollout first" strategy.
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011767.html
In which Maciej says that is polite (but not mandatory) to ask the
responsible person first.
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011768.html
In which Nikolas Zimmermann expresses concern about automatic rollouts
and asks for a 30 minute window to fix it.
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011771.html
In which Eric Seidel outlines a sheriffbot algorithm that's similar
(but slightly more agressive) than what we've built.
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011774.html
In which Jeremy Orlow supports that algorithm.
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011776.html
In which Alexey Proskuryakov asks for #webkit notification in addition
(which we did).
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011790.html
In which Maciej supports Geoffrey Garen in requesting that we build a
better failure notification system before we build an automatic
rollout system.
* https://lists.webkit.org/pipermail/webkit-dev/2010-February/011792.html
In which Maciej again asks that we have a sheriff (although I think he
had a human rather than a machine in mind):
[[
What I'd prefer to see is that the sheriff
the person primarily responsible for reverting broken patches if not
fixed in a timely manner. Then we could have some human judgment in
the process and specific people with clear responsibility.
]]

* https://lists.webkit.org/pipermail/webkit-dev/2010-March/011970.html
In which we go through the process manually on webkit-dev again.

In summary, there's been a lot of discussion on webkit-dev about how
to keep the tree green.  Several people (including senior folks in the
project) have asked us to build exactly the system we've built.  We
floated a bunch of approaches and algorithms, got feedback, and took
that feedback into account.  For example,
1) We didn't just skip the failing/flaky tests.  We spent months
fixing them.  (Ok, we might have skipped one or two, but we and a lot
of folks put a lot of effort into fixing them.)
2) Sheriffbot integrates with IRC instead of just with bugs.webkit.org.
3) Sheriffbot doesn't roll out patches automatically.  Instead, human
judgement is involved.
4) There's a minimum of process (e.g., no pre-set "fix or be rolled
out" deadline).
5) There's no notion of a "closed tree".

Of course, Eric and I wrote the majority of the code and so got to
choose various details of the how things work.  If other people are
passionate about this topic, we're open to suggestion and
contributions.  All the code is in svn.webkit.org waiting to be
improved.

Adam