[webkit-dev] Build system update

[Mark Rowe]

On 2011-03-23, at 13:49, Adam Barth wrote:

> On Wed, Mar 23, 2011 at 1:33 PM, David Levin <levin at google.com> wrote:
>> On Wed, Mar 23, 2011 at 12:17 PM, Adam Barth <abarth at webkit.org> wrote:
>>> $ time git svn rebase
>>> [... update my working copy from changes during lunch (four revisions)
>>> ...]
>>> real    1m10.316s
>>> user    0m8.194s
>>> sys     0m16.400s
>>> 
>>> $ time ./Tools/Scripts/generate-project-files
>>> real    0m4.339s
>>> user    0m3.888s
>>> sys     0m0.269s
>>> 
>>> which is about 6.1% overhead on an almost trivial update.  We can also
>>> reduce this overhead to zero in many cases by detecting that we don't
>>> need to re-generate the project files if the inputs to the generation
>>> process haven't changed.
>> 
>> It looks like you are using a slow mechanism for syncing. :)
>> TLDR version: 8% overhead for updating 191 revisions. 49% overhead for
>> trivial sync (3 revisions) -- in addition to the added complexity (when I
>> sync or switch branches in git).
>> Here's my results for updating 191 revisions (81605 through 81796).
>> 
>> $ time git fetch && time git svn rebase
>> 
>> fetch, svn rebase times
>> real 0m21.071s, 0m36.195s
>> user 0m2.271s, 0m3.205s
>> sys 0m0.497s, 0m9.428s
>> total 57 seconds for a nontrivial update.
>> $ time ./Tools/Scripts/generate-project-files
>> real 0m4.642s
>> user 0m4.243s
>> sys 0m0.322s
>> 
>> An 8% overhead on a non-trivial sync.
>> 
>> For a trivial update of 3 revisions:
>> real 0m3.490s, 0m6.017s
>> user 0m0.932s, 0m2.266s
>> sys 0m0.184s, 0m2.824s
>> For a total of 9.5 seconds.
>> 
>> The generate project files step remained the same, so this adds a 49%
>> overhead for a trivial sync.
>> It also adds time (and complications) whenever I switch branches in git.
>> dave
>> PS For chromium, my experience is
>> $ time gclient runhooks --force
>> real 0m42.459s
>> user 0m29.236s
>> sys 0m2.543s
>> I don't know enough about chromium build system and gyp to know why it is an
>> order of magnitude slower, but this overhead is noticeable and annoying
>> there. I love the idea of a one project file system, but I have concerns
>> about project file generation on sync becoming the norm in WebKit. fwiw, I
>> checked how many gyp files were in chromium and it appeared within 20%.
> 
> There seem to be four approaches to improving this situation:
> 
> 1) Check in the generated project files.  In this approach, only one
> person pays the cost of generating the project files.
> 
> 2) Make generate-project-files run faster.  I haven't looked at what's
> taking 4 seconds, but it seems entirely possible that we can improve
> the performance.
> 
> 3) Avoid running generate-project-files when not necessary.  This
> approach would improve the performance of the trivial sync, for
> example.
> 
> 4) Run generate-project-files at time other than sync.  For example,
> we could run generate-project-files as part of build-webkit, which
> already takes more more than 4 seconds, leading to a lower percentage
> overhead.

People build much more frequently than the update, so 4 would have to be done in conjunction with 2 and/or 3.

- Mark