[webkit-dev] Build system update

[Adam Barth]

On Wed, Mar 23, 2011 at 1:58 PM, Mark Rowe <mrowe at apple.com> wrote:
> On 2011-03-23, at 13:49, Adam Barth wrote:
>> On Wed, Mar 23, 2011 at 1:33 PM, David Levin <levin at google.com> wrote:
>>> On Wed, Mar 23, 2011 at 12:17 PM, Adam Barth <abarth at webkit.org> wrote:
>>>> $ time git svn rebase
>>>> [... update my working copy from changes during lunch (four revisions)
>>>> ...]
>>>> real    1m10.316s
>>>> user    0m8.194s
>>>> sys     0m16.400s
>>>>
>>>> $ time ./Tools/Scripts/generate-project-files
>>>> real    0m4.339s
>>>> user    0m3.888s
>>>> sys     0m0.269s
>>>>
>>>> which is about 6.1% overhead on an almost trivial update.  We can also
>>>> reduce this overhead to zero in many cases by detecting that we don't
>>>> need to re-generate the project files if the inputs to the generation
>>>> process haven't changed.
>>>
>>> It looks like you are using a slow mechanism for syncing. :)
>>> TLDR version: 8% overhead for updating 191 revisions. 49% overhead for
>>> trivial sync (3 revisions) -- in addition to the added complexity (when I
>>> sync or switch branches in git).
>>> Here's my results for updating 191 revisions (81605 through 81796).
>>>
>>> $ time git fetch && time git svn rebase
>>>
>>> fetch, svn rebase times
>>> real 0m21.071s, 0m36.195s
>>> user 0m2.271s, 0m3.205s
>>> sys 0m0.497s, 0m9.428s
>>> total 57 seconds for a nontrivial update.
>>> $ time ./Tools/Scripts/generate-project-files
>>> real 0m4.642s
>>> user 0m4.243s
>>> sys 0m0.322s
>>>
>>> An 8% overhead on a non-trivial sync.
>>>
>>> For a trivial update of 3 revisions:
>>> real 0m3.490s, 0m6.017s
>>> user 0m0.932s, 0m2.266s
>>> sys 0m0.184s, 0m2.824s
>>> For a total of 9.5 seconds.
>>>
>>> The generate project files step remained the same, so this adds a 49%
>>> overhead for a trivial sync.
>>> It also adds time (and complications) whenever I switch branches in git.
>>> dave
>>> PS For chromium, my experience is
>>> $ time gclient runhooks --force
>>> real 0m42.459s
>>> user 0m29.236s
>>> sys 0m2.543s
>>> I don't know enough about chromium build system and gyp to know why it is an
>>> order of magnitude slower, but this overhead is noticeable and annoying
>>> there. I love the idea of a one project file system, but I have concerns
>>> about project file generation on sync becoming the norm in WebKit. fwiw, I
>>> checked how many gyp files were in chromium and it appeared within 20%.
>>
>> There seem to be four approaches to improving this situation:
>>
>> 1) Check in the generated project files.  In this approach, only one
>> person pays the cost of generating the project files.
>>
>> 2) Make generate-project-files run faster.  I haven't looked at what's
>> taking 4 seconds, but it seems entirely possible that we can improve
>> the performance.
>>
>> 3) Avoid running generate-project-files when not necessary.  This
>> approach would improve the performance of the trivial sync, for
>> example.
>>
>> 4) Run generate-project-files at time other than sync.  For example,
>> we could run generate-project-files as part of build-webkit, which
>> already takes more more than 4 seconds, leading to a lower percentage
>> overhead.
>
> People build much more frequently than the update, so 4 would have to be done in conjunction with 2 and/or 3.

Indeed.  I suspect (2) and (3) are worth doing regardless.

Adam