[webkit-dev] git.webkit.org problem

[Lucas Forschler]

Hi Peter,

I am looking into improving performance of build.webkit.org.  After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests.  I believe requests are processed serially, and not in parallel.  The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers.  As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity.  We may have reached a limit on the number of slaves we can handle with our current configuration.  

Some of the things I have been researching to increase build.webkit.org performance include the following:

-Upgrades to hardware.  It's possible faster hardware could alleviate some problems.  It is unclear if this is really the limiting factor, or how much additional capacity this would provide.
-Updating the way files are transferred
	(https://bugs.webkit.org/show_bug.cgi?id=73484)
-Upgrade the version of buildbot. (0.8.3 -> 0.8.5).  
	Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related. 
-Upgrading the version of twisted running on the sever.  We are currently at version 8.2, and version 12 is now available.
-Investigating some kind of apache caching layer on top of buildbot
-Adding a second master to the buildbot system.
	buildbot supports multi-master mode.  We could potentially split out build and test onto two separate masters.
	build.webkit.org and test.webkit.org ?

I would be interested in hearing any other thoughts on increasing build.webkit.org performance.  

Lucas


On Feb 29, 2012, at 7:08 AM, Peter Beverloo wrote:

> Hi Bill,
> 
> Would it be possible to get some more information on your/Lucas' plans on improving stability of the infrastructure?  This was the third significant interruption within a week, and especially for us non-PST folks it's becoming quite inconvenient.
> 
> Thanks,
> Peter
> 
> On Wed, Feb 29, 2012 at 15:00, William Siegrist <wsiegrist at apple.com> wrote:
> This should be fixed now.
> 
> -Bill
> 
> On Feb 29, 2012, at 3:19 AM, David Barr wrote:
> 
> >> Is it any trick to update my git-svn repo from svn?
> >>
> >> I set up my git svn repo about
> >> $http://trac.webkit.org/wiki/UsingGitWithWebKit :
> >> - git clone git://git.webkit.org/WebKit.git WebKit
> >> - git svn init --prefix=origin/ -T trunk
> >> http://svn.webkit.org/repository/webkit
> >> - git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master
> >
> > Usually, the only missing step here is 'git svn fetch'.
> > However, when I run that now I get:
> >
> > RA layer request failed: OPTIONS of
> > 'http://svn.webkit.org/repository/webkit': could not connect to server
> > (http://svn.webkit.org) at /Users/davidbarr/libexec/git-core/git-svn
> > line 2139
> >
> > --
> > David Barr.
> > _______________________________________________
> > webkit-dev mailing list
> > webkit-dev at lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120229/9d2d7ef4/attachment.html>