[webkit-dev] Common build system (was Re: WebKit Wishes)

[Nico Weber]

On Sat, Feb 2, 2013 at 4:58 PM, Adam Barth <abarth at webkit.org> wrote:
> Ninja has extremely fast incremental builds and can be generated by
> GYP.  Here are some stats from a year ago:
>
> https://plus.google.com/101038813433650812235/posts/irc26fhRtPC
>
> Ninja has gotten even faster since then.  If you're interested in
> trying it out, you can play around with incremental builds of the
> Chromium port on Mac or Linux.

You can also look at the build output from the chromium bots.

Empty build in 1s:
http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66807/steps/compile-webkit/logs/stdio
Build with a few files changed in 15s:
http://build.webkit.org/builders/Chromium%20Linux%20Release/builds/66800/steps/compile-webkit/logs/stdio

…and this is on fairly slow bots. On my SSD-equipped laptop, I can do
incremental rebuilds of all of chrome after touching one (cpp or mm)
file in 2-6s.

>
> Adam
>
>
> On Fri, Feb 1, 2013 at 4:58 PM, Balazs Kelemen <kbalazs at webkit.org> wrote:
>> I think one important aspect of build systems was not considered yet int
>> this conversation: speed. The time an incremental build takes has a great
>> effect on developer productivity. I don't think any of the meta-build
>> systems we use does a great job here - although I only have experiences with
>> qmake, cmake and autotools (and I don't have an SSD that could help
>> somewhat).
>>
>> The technic I found useful here is to avoid calling build-webkit always and
>> instead just rebuild the subproject you have edited, so I think it is
>> important to have a build system that supports it. Let me share my
>> experiences here.
>>
>> With qmake nowadays this work perfectly. The developer build is producing a
>> shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2),
>> which means you only need to call make in the specific subdirectory (i.e. if
>> I only touched WebKit2 files I do "make -C
>> WebKitBuild/Release/Source/WebKit2" which is pretty quick). Still WebCore is
>> so big that make is quite slowly find out the files you actually edited and
>> need to be rebuilt. What could help here is to devide WebCore into smaller
>> parts, like the ongoing work of extracting Platform. Maybe the next logical
>> candidate could be svg (I don't have real knowledge about how feasible it
>> is).
>>
>> Note that I don't come up with qmake because I would like to recommend it as
>> the one and only build system (in fact it has a ridiculously inconvenient
>> syntax, and a lot of bugs), just as an example.
>>
>> With Cmake fast incremental rebuilds are also possible, maybe in a bit more
>> complicated way. When working with the EFL port I found a quick rule for
>> WebKit2 in the generated makefile. Although the makefiles are usually call
>> back to Cmake, and make is not faster than build-webkit, if you use the
>> special fast target, which is something like eflWebKit/fast (i.e. make -C
>> WebKitBuild/Release/Source/WebKit2 eflWebKit/fast), it will not check
>> dependencies but just rebuild the files that have changed. I did not find a
>> similar thing for WebCore, I guess because it is not built as a shared
>> library.
>>
>> What I dislike in Cmake is that I am disappointed by how slow a normal
>> incremental build with it (i.e. build-webkit). qmake is not faster at all,
>> but it generate plain makefiles that typically no call back to qmake if not
>> specified explicitly to do so, and directly calling make is faster, yet it
>> can handle most of the non-trivial changes, for example editing a generator
>> file. I don't know gyp, so I wonder about how would it do in this comparison
>> (but I guess it generates simple makefiles as well, so it's similar than
>> qmake in this manner.)
>>
>> I hope I added something to this conversation that is worth to consider with
>> my late nightly brain dump.
>>
>> -kbalazs
>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
> On Fri, Feb 1, 2013 at 4:58 PM, Balazs Kelemen <kbalazs at webkit.org> wrote:
>> I think one important aspect of build systems was not considered yet int
>> this conversation: speed. The time an incremental build takes has a great
>> effect on developer productivity. I don't think any of the meta-build
>> systems we use does a great job here - although I only have experiences with
>> qmake, cmake and autotools (and I don't have an SSD that could help
>> somewhat).
>>
>> The technic I found useful here is to avoid calling build-webkit always and
>> instead just rebuild the subproject you have edited, so I think it is
>> important to have a build system that supports it. Let me share my
>> experiences here.
>>
>> With qmake nowadays this work perfectly. The developer build is producing a
>> shared library for every subdir (WTF, JavaScriptCore, WebCore, WebKit2),
>> which means you only need to call make in the specific subdirectory (i.e. if
>> I only touched WebKit2 files I do "make -C
>> WebKitBuild/Release/Source/WebKit2" which is pretty quick). Still WebCore is
>> so big that make is quite slowly find out the files you actually edited and
>> need to be rebuilt. What could help here is to devide WebCore into smaller
>> parts, like the ongoing work of extracting Platform. Maybe the next logical
>> candidate could be svg (I don't have real knowledge about how feasible it
>> is).
>>
>> Note that I don't come up with qmake because I would like to recommend it as
>> the one and only build system (in fact it has a ridiculously inconvenient
>> syntax, and a lot of bugs), just as an example.
>>
>> With Cmake fast incremental rebuilds are also possible, maybe in a bit more
>> complicated way. When working with the EFL port I found a quick rule for
>> WebKit2 in the generated makefile. Although the makefiles are usually call
>> back to Cmake, and make is not faster than build-webkit, if you use the
>> special fast target, which is something like eflWebKit/fast (i.e. make -C
>> WebKitBuild/Release/Source/WebKit2 eflWebKit/fast), it will not check
>> dependencies but just rebuild the files that have changed. I did not find a
>> similar thing for WebCore, I guess because it is not built as a shared
>> library.
>>
>> What I dislike in Cmake is that I am disappointed by how slow a normal
>> incremental build with it (i.e. build-webkit). qmake is not faster at all,
>> but it generate plain makefiles that typically no call back to qmake if not
>> specified explicitly to do so, and directly calling make is faster, yet it
>> can handle most of the non-trivial changes, for example editing a generator
>> file. I don't know gyp, so I wonder about how would it do in this comparison
>> (but I guess it generates simple makefiles as well, so it's similar than
>> qmake in this manner.)
>>
>> I hope I added something to this conversation that is worth to consider with
>> my late nightly brain dump.
>>
>> -kbalazs
>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev