[webkit-dev] git.webkit.org problem

[Marc-Antoine Ruel]

Here are some things that may help;

*#1 svn server overloading when all the slaves bootstrap at the same time*
The only good work around is to keep an internal read-only svn-mirror that
the slaves use instead of the real svn server.

This means that builds must be triggered on the svn-mirror, which may
incurs a delay of a few seconds, due to svnsync polling. My hypothesis is
that svn has repo-wide locks while committing, so when there are a lot of
readers, they are all starved whenever some action is happening. Doing the
svn-mirror trick helped us to scale in the range past *several hundreds of
simultaneous active connections*. As a guidance, gclient sync does 8 svn
connections per slave by default. Scale that to hundreds of slaves waking
up at the same time...

Also, serving the read-only svn server from a set of apache frontends
helps. We do that with src.chromium.org. To be clear, above I was talking
about a DMZ-only mirror. It is a different one from the apache frontends.
Implementing both has the biggest benefit.


*#2 Using Chromium's git svn clone of webkit*
We have a few read only git frontends that are relatively speedy. I just
verified and both master at HEAD matches:
git remote add chromium http://git.chromium.org/external/Webkit.git
git fetch chromium

Once you verified that the hash matches locally with:
git fetch --all
git log -1 origin/master
git log -1 chromium/master

Then you may want to;
git config remote.chromium.fetch +refs/heads/*:refs/remotes/origin/*
git branch -r -D chromium/master

I give no guarantee what-so-ever about freshness, lack of corruption,
consistent performance, availability, YMMV, etc.
Note the url has a lower K!

Note that we also have WebKit_trimmed.git (note the upper case K!) that is
much smaller due to removal of many files chromium developers don't need
but it has invalid hashes so *do not fetch this one in your current webkit
git-svn clone*.


*#3 Buildbot's (absence of) performance*
Stefan (cc'ed) is also looking at this and has been profiling our slowest
masters for a while. In our case, we are CPU-bound and it looks like we
have to tune our caching. Stefan can explain it better than me. You guys
probably want to talk together.


Hope this is useful,

M-A


Le 29 février 2012 21:50, William Siegrist <wsiegrist at apple.com> a écrit :

> We're back to the slaves overloading svn. The Apple slaves are blocked
> temporarily while everyone else catches up. Sorry for the inconvenience.
>
> -Bill
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120301/e2c168eb/attachment-0001.html>