[webkit-dev] git.webkit.org problem

[Mark Rowe]

On 2012-02-29, at 17:05, Lucas Forschler <lforschler at apple.com> wrote:

> build.webkit.org should be back online now with 0.8.5.
> 
> Thanks for your patience!

For anyone following along at home, upgrading to Buildbot v0.8.5 alone actually made build.webkit.org substantially slower. After profiling it for a while we noticed that it was spending an exceptionally long time performing queries against its SQLite database. Some analysis on the queries that were being run showed that the database lacked appropriate indices for the queries, causing full table scans across rather large tables. Ouch. A few "CREATE INDEX …" statements later and now build.webkit.org is responding much more quickly than it has been in quite some time.

If anyone notices any bad behavior that they suspect may be related to this upgrade, please let us know.

- Mark.

> Lucas
> 
> On Feb 29, 2012, at 3:44 PM, Lucas Forschler wrote:
> 
>> I'm currenlty upgrading build.webkit.org from 0.8.3 to 0.8.5.  It should be back online shortly.  
>> Lucas
>> 
>> On Feb 29, 2012, at 12:38 PM, Ryosuke Niwa wrote:
>> 
>>> On Wed, Feb 29, 2012 at 11:32 AM, Lucas Forschler <lforschler at apple.com> wrote:
>>> I am looking into improving performance of build.webkit.org.  After doing some initial research I feel that buildbot isn't the most efficient in terms of processing a large number of requests.  I believe requests are processed serially, and not in parallel.  The buildbot documentation mentions a limit to the number of slaves that can be supported, but doesn't give any specific numbers.  As webkit has grown, and the number of builds and slaves we are supporting has increased, it seems the server is showing signs of reaching this capacity.  We may have reached a limit on the number of slaves we can handle with our current configuration.
>>> 
>>> On Chromium side, buildbot appears to leak a great amount of memory. e.g. we end up running out of memory at some point because buildbot keeps leaking some memory over time :(  Do you see the same issue on your buildbot?
>>> 
>>> -Upgrades to hardware.  It's possible faster hardware could alleviate some problems.  It is unclear if this is really the limiting factor, or how much additional capacity this would provide.
>>> 
>>> My understanding is that we use a really fast server but still experience a similar problem.
>>>  
>>> -Updating the way files are transferred
>>> 	(https://bugs.webkit.org/show_bug.cgi?id=73484)
>>> 
>>> Would it make sense to have a separate apache server, etc... for receiving these files? twistd doesn't appear to be a cut out for uploading/downloading megabytes of data.
>>> 
>>> -Upgrade the version of buildbot. (0.8.3 -> 0.8.5).  
>>> 	Version 0.8.4 and higher supposedly has a threading improvement, but it's unclear how much it would benefit us since it looks to be mostly database related.
>>> 
>>> I've been told that upgrading buildbot didn't improve the performance. But upgrading it to 0.8.5 is probably a good nonetheless since it fixes various bugs I've encountered while maintaining Chromium bots and also when I ran buildbot locally.
>>> 
>>> -Investigating some kind of apache caching layer on top of buildbot
>>> 
>>> MUST DO THIS. This will greatly reduce the latency for most people.
>>> 
>>> -Adding a second master to the buildbot system.
>>> 	buildbot supports multi-master mode.  We could potentially split out build and test onto two separate masters.
>>> 	build.webkit.org and test.webkit.org ?
>>> 
>>> build.chromium.org takes this approach. But then we won't be able to see both build and test step at once though... Would it make more sense to split per port? e.g. Apple Mac & Apple Win, Qt & Gtk, Chromium & misc or something along that line?
>>> 
>>> - Ryosuke
>>> 
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120229/7212670b/attachment.html>