[webkit-dev] Build system update

[Adam Barth]

On Wed, Mar 23, 2011 at 1:33 PM, David Levin <levin at google.com> wrote:
> On Wed, Mar 23, 2011 at 12:17 PM, Adam Barth <abarth at webkit.org> wrote:
>> $ time git svn rebase
>> [... update my working copy from changes during lunch (four revisions)
>> ...]
>> real    1m10.316s
>> user    0m8.194s
>> sys     0m16.400s
>>
>> $ time ./Tools/Scripts/generate-project-files
>> real    0m4.339s
>> user    0m3.888s
>> sys     0m0.269s
>>
>> which is about 6.1% overhead on an almost trivial update.  We can also
>> reduce this overhead to zero in many cases by detecting that we don't
>> need to re-generate the project files if the inputs to the generation
>> process haven't changed.
>
> It looks like you are using a slow mechanism for syncing. :)
> TLDR version: 8% overhead for updating 191 revisions. 49% overhead for
> trivial sync (3 revisions) -- in addition to the added complexity (when I
> sync or switch branches in git).
> Here's my results for updating 191 revisions (81605 through 81796).
>
> $ time git fetch && time git svn rebase
>
> fetch, svn rebase times
> real 0m21.071s, 0m36.195s
> user 0m2.271s, 0m3.205s
> sys 0m0.497s, 0m9.428s
> total 57 seconds for a nontrivial update.
> $ time ./Tools/Scripts/generate-project-files
> real 0m4.642s
> user 0m4.243s
> sys 0m0.322s
>
> An 8% overhead on a non-trivial sync.
>
> For a trivial update of 3 revisions:
> real 0m3.490s, 0m6.017s
> user 0m0.932s, 0m2.266s
> sys 0m0.184s, 0m2.824s
> For a total of 9.5 seconds.
>
> The generate project files step remained the same, so this adds a 49%
> overhead for a trivial sync.
> It also adds time (and complications) whenever I switch branches in git.
> dave
> PS For chromium, my experience is
> $ time gclient runhooks --force
> real 0m42.459s
> user 0m29.236s
> sys 0m2.543s
> I don't know enough about chromium build system and gyp to know why it is an
> order of magnitude slower, but this overhead is noticeable and annoying
> there. I love the idea of a one project file system, but I have concerns
> about project file generation on sync becoming the norm in WebKit. fwiw, I
> checked how many gyp files were in chromium and it appeared within 20%.

There seem to be four approaches to improving this situation:

1) Check in the generated project files.  In this approach, only one
person pays the cost of generating the project files.

2) Make generate-project-files run faster.  I haven't looked at what's
taking 4 seconds, but it seems entirely possible that we can improve
the performance.

3) Avoid running generate-project-files when not necessary.  This
approach would improve the performance of the trivial sync, for
example.

4) Run generate-project-files at time other than sync.  For example,
we could run generate-project-files as part of build-webkit, which
already takes more more than 4 seconds, leading to a lower percentage
overhead.

Adam