[webkit-dev] Yet another email about a broken tree

[Maciej Stachowiak]

On Mar 17, 2010, at 1:49 AM, Kenneth Russell wrote:

> On Wed, Mar 17, 2010 at 12:44 AM, Maciej Stachowiak <mjs at apple.com>  
> wrote:
>>
>> On Mar 17, 2010, at 12:02 AM, Adam Barth wrote:
>>
>>>
>>> Manual investigation seems to implicate
>>> <http://trac.webkit.org/changeset/56074> for at least some of the
>>> brokenness in SnowLeopard Intel Release (Tests).  There's a patch in
>>> <https://bugs.webkit.org/show_bug.cgi?id=36194> that claims to fix
>>> things, but the brokenness has been polluting the tree for 12 hours.
>>> Is there some reason we didn't roll out the offending patch?
>>
>> Kenneth intended to temporarily disable the failing tests:
>> <http://trac.webkit.org/changeset/56093>. That seems to have  
>> cleaned up the
>> failures on other bots, but there is still a great number of  
>> failures on
>> SnowLeopard release.  I'm not sure what we can do other than revert  
>> the
>> patch until the correct fix is ready.
>
> I apologize for the widespread WebGL test breakage from this patch. I
> sidestepped the commit queue because other broken tests had blocked
> it, and this patch was blocking other work. While the patch worked
> fine on our local machines (both Leopard and Snow Leopard), it seems
> that OpenGL driver bugs on the build bots caused many tests to fail,
> in particular on the Snow Leopard bot. This afternoon we skipped the
> tests that were blocking the bots the cq cared about.

I don't think it's that great to leave breakage on some of the bots  
solely because the commit queue doesn't pay attention to them.

(BTW do we know if the change might tickle similar driver bugs on any  
other hardware/software combos that we haven't tested yet? Do we have  
a plan for finding out? Has anyone reported the relevant driver bugs  
to the appropriate vendor?)

> Mo worked as quickly as possible today to come up with a workaround
> for the broken drivers, and a patch is out for review, but perhaps the
> original patch should be rolled out after all, and a revised version
> re-committed through the queue.

If it's going to be more than a couple of hours before the proper fix  
can land, then I would recommend that.

>
> Again, I apologize for the breakage. It would be best for everyone, I
> think, if we got the tree to a green state and all committed through
> the queue, thereby having a line of defense against unexpected test
> failures on the bots.

If we really want everyone to use the commit queue for most normal  
work, we really have to fix it so that it puts a meaningful value in  
SVN's committer field.

That being said, the mechanism I'd really like to see first is better  
notification of when the bot goes red (I suspect a number of people  
involved in today's redness didn't notice right away because there is  
no active notification system).

Regards,
Maciej