[webkit-dev] On the greeness of the GTK+ bot

[Xan Lopez]

On Mon, Nov 16, 2009 at 3:33 PM, Gustavo Noronha Silva <gns at gnome.org> wrote:
> On Mon, 2009-11-16 at 05:24 -0800, Adam Barth wrote:
>> Eric and I are working on a bot that might help this situation.
>> Essentially, the bot will try out patches on Qt and GTK and add a
>> comment to the bug if the patch regresses the build.  Our plan is to
>> start with compiling, but we'd eventually like to run the tests as
>> well.
>
> That sounds like an awesome idea. Thanks for working on it. Build
> breakages themselves have become less of an issue for us in recent times
> - people are definitely more aware of our bots, and are acting on fixing
> them when they break.
>
> I think such an automated approach to running the build, and tests for
> upcoming patches will surely help with giving this a second step
> forward.

This is nice to see, but as Gustavo says build breakage is not too
much an issue at the moment for us: the build does not break very
often, and when it does people usually take the time to figure out
what happened and either do fix it themselves or poke us about it.
That being said, this could be improved in any number of ways and I'm
happy to see it getting ever better.

What is effectively a black hole with respect to our time is the tests
breakage, though. We get new tests failing very regularly (either
through new tests or through new code making old tests fail), and once
the bots are red people tend to pay even less attention to them, so
they spiral out of control in a positive feedback loop until we have
tests failing in the double digits in a matter of days (or hours!). Of
course in an ideal world we'd have a team big enough to always have at
least one person looking at this and fixing the problems the moment
they arise, but unfortunately this is not the case.

I realize this is a very complicated problem to fix unless we throw
more people at it from our side, but maybe we could all agree in some
basic guidelines to make things a bit better:

- If you are adding new functionality + tests that you *know* will
fail in some bots, open a bug describing what we should implement to
get it working, and skip all new tests with a reference to the bug
number. This should be relatively easy to do for the person who
implemented the functionality in the first place, and would save
everyone lots of time in the long run.

- If your new code makes a previously passing test(s) fail in a port
other than your own, you have two choices:
  * If you know what's going on and think this is a bug in the port,
you can propose a patch yourself, or open a bug with your ideas and
skip the tests with a reference to the bug number. Get the port
maintainers on the loop ASAP.
  * If you have no idea of what's going on, maybe Eric's proposed
hard-line policy kicks in: you either figure it out quick and propose
something, or the patch is rolled out until things are clear;
regressions in tests in any port are as bad as build breakages, so
early roll out is preferred to having the bots red for a long time.

As I said, this is a complex problem, but I think at the very least
having a clear picture of what is expected of everyone with respect to
failing tests in the bots would be a good start to make things easier.

Xan

>
> See you,
>
> --
> Gustavo Noronha Silva <gns at gnome.org>
> GNOME Project
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>