[webkit-dev] On the greeness of the GTK+ bot

[Andras Becsi]

Hello,
for us too, as many of you poited out before me, the main issue is not 
build breakage while maintaining the QtWebKit buildbot - because these 
are most of the time trivial fixes - but tracking down the previously 
passing but after-the-night failing tests.
Because of the timezone differences most of the work on the Mac port is 
done when we here in Europe are in our sleeping-quarters, and a green 
bot will most certainly be red in the morning and a non-green bot gets 
out of sight very quickly.
However, I find the idea of the try-bots great, because this takes the 
right direction to a future where try-bots also test the layout tests.
We are monitoring changes and their effect on our buildbot and tracking 
down incremental failures and flakey tests which cause false alarm and 
disrupt the trustiness of our bot.
Another huge problem is, as Dimitri pointed out too, that in most cases 
we miss a certain DRT feature to run a test correctly. If we spend more 
effort improving (and btw fixing) DRT and the test infrastructure than 
improving and fixing the port itself then there might be a problem with 
the layout-testing approach on multiple ports.

BR:
Andras (bbandix)

Dimitri Glazkov írta:
> On Mon, Nov 16, 2009 at 5:56 AM, Xan Lopez <xan at gnome.org> wrote:
>> On Mon, Nov 16, 2009 at 3:33 PM, Gustavo Noronha Silva <gns at gnome.org> wrote:
>>> On Mon, 2009-11-16 at 05:24 -0800, Adam Barth wrote:
>>>> Eric and I are working on a bot that might help this situation.
>>>> Essentially, the bot will try out patches on Qt and GTK and add a
>>>> comment to the bug if the patch regresses the build.  Our plan is to
>>>> start with compiling, but we'd eventually like to run the tests as
>>>> well.
>>> That sounds like an awesome idea. Thanks for working on it. Build
>>> breakages themselves have become less of an issue for us in recent times
>>> - people are definitely more aware of our bots, and are acting on fixing
>>> them when they break.
>>>
>>> I think such an automated approach to running the build, and tests for
>>> upcoming patches will surely help with giving this a second step
>>> forward.
>> This is nice to see, but as Gustavo says build breakage is not too
>> much an issue at the moment for us: the build does not break very
>> often, and when it does people usually take the time to figure out
>> what happened and either do fix it themselves or poke us about it.
>> That being said, this could be improved in any number of ways and I'm
>> happy to see it getting ever better.
>>
>> What is effectively a black hole with respect to our time is the tests
>> breakage, though. We get new tests failing very regularly (either
>> through new tests or through new code making old tests fail), and once
>> the bots are red people tend to pay even less attention to them, so
>> they spiral out of control in a positive feedback loop until we have
>> tests failing in the double digits in a matter of days (or hours!). Of
>> course in an ideal world we'd have a team big enough to always have at
>> least one person looking at this and fixing the problems the moment
>> they arise, but unfortunately this is not the case.
> 
> This is a huge issue with the Chromium port as well. We spend quite a
> bit of effort tracking down failing tests, only to discover that the
> failure is due to one-port baselines or new functionality added to
> DRT. I wonder if the approach we have today in regards to tests is not
> sustainable with multiple vibrant ports, each spending way to much
> time catching up.
> 
> :DG<
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev