[webkit-dev] Yet another email about a broken tree

[Kenneth Russell]

On Wed, Mar 17, 2010 at 2:16 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>
> On Mar 17, 2010, at 1:49 AM, Kenneth Russell wrote:
>
>> On Wed, Mar 17, 2010 at 12:44 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>>>
>>> On Mar 17, 2010, at 12:02 AM, Adam Barth wrote:
>>>
>>>>
>>>> Manual investigation seems to implicate
>>>> <http://trac.webkit.org/changeset/56074> for at least some of the
>>>> brokenness in SnowLeopard Intel Release (Tests).  There's a patch in
>>>> <https://bugs.webkit.org/show_bug.cgi?id=36194> that claims to fix
>>>> things, but the brokenness has been polluting the tree for 12 hours.
>>>> Is there some reason we didn't roll out the offending patch?
>>>
>>> Kenneth intended to temporarily disable the failing tests:
>>> <http://trac.webkit.org/changeset/56093>. That seems to have cleaned up
>>> the
>>> failures on other bots, but there is still a great number of failures on
>>> SnowLeopard release.  I'm not sure what we can do other than revert the
>>> patch until the correct fix is ready.
>>
>> I apologize for the widespread WebGL test breakage from this patch. I
>> sidestepped the commit queue because other broken tests had blocked
>> it, and this patch was blocking other work. While the patch worked
>> fine on our local machines (both Leopard and Snow Leopard), it seems
>> that OpenGL driver bugs on the build bots caused many tests to fail,
>> in particular on the Snow Leopard bot. This afternoon we skipped the
>> tests that were blocking the bots the cq cared about.
>
> I don't think it's that great to leave breakage on some of the bots solely
> because the commit queue doesn't pay attention to them.
>
> (BTW do we know if the change might tickle similar driver bugs on any other
> hardware/software combos that we haven't tested yet? Do we have a plan for
> finding out? Has anyone reported the relevant driver bugs to the appropriate
> vendor?)

Our best current plan is more widespread testing. We will file a Radar
bug as soon as we have more information about the nature of the
failure -- by virtue of working around the bugs. If we knew the
precise hardware configuration of the bots, including graphics cards,
that would help.

>> Mo worked as quickly as possible today to come up with a workaround
>> for the broken drivers, and a patch is out for review, but perhaps the
>> original patch should be rolled out after all, and a revised version
>> re-committed through the queue.
>
> If it's going to be more than a couple of hours before the proper fix can
> land, then I would recommend that.

I'm afraid I personally can't do anything at this point until probably
9 or 10 AM Pacific. If someone wants to revert the patch in the
interim, that's fine. I hope that we will have the proper workaround
for the driver bugs by early afternoon Pacific.

>> Again, I apologize for the breakage. It would be best for everyone, I
>> think, if we got the tree to a green state and all committed through
>> the queue, thereby having a line of defense against unexpected test
>> failures on the bots.
>
> If we really want everyone to use the commit queue for most normal work, we
> really have to fix it so that it puts a meaningful value in SVN's committer
> field.
>
> That being said, the mechanism I'd really like to see first is better
> notification of when the bot goes red (I suspect a number of people involved
> in today's redness didn't notice right away because there is no active
> notification system).

For what it's worth, the tree was red before I committed due to at
least two or three flaky tests which have been showing up for days. An
active notification system would not have helped here. I think what
should be done is to get the tree green by skipping these flaky tests;
file high-priority bugs against the test authors to fix the flakiness;
and then figure out a way the commit queue can be used for the vast
majority of patches. (A secure repository of committer
username/passwords on the machine actually executing the commit?)

-Ken

> Regards,
> Maciej
>
>