[webkit-dev] svn.webkit.org

[Osztrogonac Csaba]

Mark Rowe írta:
> On 2012-03-01, at 03:37, Osztrogonac Csaba <oszi at inf.u-szeged.hu> wrote:
>> After unbanning our network, I'll copy a locally tar-ed WebKit-svn
>> copy to all bots and then restart them one by one not to overload
>> svn.webkit.org.
> 
> You can get a relatively up to date working copy from <http://nightly.webkit.org/files/WebKit-SVN-source.tar.bz2>.

We always have relatively up-to-date working copy. It is faster then downloading
from you. ;) (I tried to download this nightly and the ETA is ~3 hours ...)
Additionally we use svn 1.7 (to make our bots faster, and less IO intensive)
which has different working copy than svn 1.6 .

>> And I'm thinking about how can we make buildbots more robust in the future.
>> My first idea is that we should setup git mirrors for all WebKit port, and
>> then make buildbots use these mirrors instead of svn.webkit.org. To do it,
>> we need to modify the master.cfg a little bit.
> 
> There are two different options that we're investigating to address this "thundering herd" problem that tends to kill SVN after buildbot downtime:
> 1) Teach build slaves to fetch and unpack <http://nightly.webkit.org/files/WebKit-SVN-source.tar.bz2> rather than using "svn checkout".

To do it, all slave must use same svn version.

> 2) Add additional hardware and load-balance svn.webkit.org across several machines so that the spike in load is distributed.

It is a good idea. ;)

>> To migrate the bots simple from svn to git isn't a good solution, because in this case
>> we'll loose the svn revision number on the waterfall. My idea is that replace the
>> "self.addStep(CheckOutSource)" to calling a shell/perl/python script which updates
>> the local copy from the Qt/GTK/Apple/Chromium/... git mirror with the following way:
>> - git fetch
>> - git reset --hard `git svn find-rev r<svn-revision-number-got-from-the-master>`
>>
>> I'm going to implement this initial git updater script this week and try to migrate our
>> bots on build.webkit.sed.hu to use it. If we manage to make it stable, we can make
>> build.webkit.org slaves to use it too. How does it sound? Any better idea?
> 
> Pulling from git instead of SVN is certainly worth considering, but I wouldn't be surprised if this simply shifted the performance issue from svn.webkit.org to git.webkit.org.

No, I don't want to shift the load from svn.webkit.org to git.webkit.org. But make all port
maintainer to use their own git mirror for the bots instead of the root svn/git.webkit.org.
Mirroring git.webkit.org locally is the simplest thing in the world. :)

br,
Ossy