[webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Maciej Stachowiak]

> On Aug 28, 2017, at 9:20 PM, Daniel Bates <dbates at webkit.org> wrote:
> 
> Thank you for looking into speeding up the build.
> 
> How does the speed gain with your proposed "unified source" approach compare to using CMake + Ninja to build currently (I think builder Apple El Capitan CMake Debug does this)?

I think unified sources help in a way that's orthogonal to what CMake+Ninja do, so we'd ideally want to apply both to a build of the whole WebKit stack.

Ideally this combination would become the default for engineering builds of the macOS and iOS ports.

> Do we know what is the cause(es) for the slow clean builds? I am assuming that much of the speed up from the "unified source" comes from clang being able to use an in-memory #include cache to avoid disk I/O and re-parsing of a seen header. Have we exhausted all efforts (or have we given up?) removing extraneous #includes? Do we think this pursuing this effort would be more time consuming or have results that pale in comparison to the "unified source" approach?

My 2c:

Probably the better long-term solution would be C++ modules once they are ready for prime time. In the short term, unified sources seem like a much faster way to get a big win than include cleanup. There's also some reason to believe unified sources can result in a higher potential win than include pruning, since we in fact have many headers that legitimately need to be included in lots of files. 


 - Maciej

> 
> 
>> On Aug 28, 2017, at 5:37 PM, Keith Miller <keith_miller at apple.com> wrote:
>> 
>> Hello fellow Webkittens,
>> 
>> Are you growing tired of long cat naps while waiting 45 minutes for WebKit to build? (I’ve definitely never done this 🤐!)
>> Do you want WebKit to build up to 3-4x faster on a clean build?*
>> Does seeing your incremental build… stay the same sound good?
>> 
>> You’ll be purring with excitement after you switch to unified source builds!**
>> 
>> * Building JSC with CMake, including CMake configuration time, went from ~8m15s to ~2m30s on my MBP using unified sources only on cpp files in JavaScriptCore. Final results on WebKit may vary. Using unified sources may cause sudden feelings of euphoria and productivity (or grumpiness from lack of naps…). Not responsible for any bugs in extra patches produced by using unified source builds (excluding build system related bugs) except where required by law (pizlo at apple.com).
>> 
>> ** Soon. Unified source builds haven’t been landed yet.
>> 
>> How are things going to change? Great question! Let’s find out!
>> 
>> When you fire off the build script it will automagically paw your source files into bundles.
> 
> How does this work if you build from Xcode? I am assuming these bundles won't interfere with debugging in Xcode. When debugging in Xcode/lldb/gdb will Xcode/lldb/gdb resolve line information with respect to individual files or the bundles?
> 
>> The size of each bundle is speculatively going to be 8 .cpp files but that’s up for experimentation. From my testing so far, the compile time for a bundle is at most 20% (~6s vs ~7s for the Air directory) longer than the longest file it includes. Given that most of the build time in incremental builds is scanning dependencies, this change is probably only barely noticeable.
>> 
>> Bundling happens as part of the CMake configuration process, in the CMake build. In the Xcode build it’s more complicated since we cannot dynamically choose the files we are going to compile. The only solution I know of is to pre-list a bunch of files for the build to dump files into. Occasionally, we’ll need to add more files to the build list.
>> 
>> When you add or remove a file in the project, the cost is higher. The current plan to bundle source files is by directory. This means that when a .cpp file is added or removed you will rebalance the bundles in its directory, and you will have to rebuild up to the full directory that file is in. 
>> 
>> My goal is to make all of this invisible to developers as much as possible. I believe, for the most part, things should operate mostly as they have before. As the details on unified source builds get finalized, I’m sending out a separate email documenting any workflow changes. I’ll also do my best to answer any questions or concerns.
>> 
>> “But Keith, am I going to have to make major changes to the way I develop WebKit?” Yes and, hopefully, no. There is one major common workflow change that comes with unified sources, the naming of static variables and functions.
> 
> I take it you mean any file-level function or definition with internal linkage. That is, this would also affect file-level constants.
> 
>> Since C++ does not have a way to only declare a static function or variable in some restricted top level scope we are going to need to create some work around. The choice I’m thinking we’ll go with is to have an auto-generated macro value FILENAME, which is similar to __FILE__ except that it doesn’t have file extension (We can’t use FILE since that’s actually a type in C 🙁). This macro would be redefined in the unified source directly above the file that is about to be included. Then in each cpp file that has static functions/variables developers would have:
>> 
>> namespace FILENAME {
>> 	static int myVariable { 0 };
>> 	static int myFunction() { return myVariable; }
>> }
>> 
>> The advantage of using a macro over textually writing the filename is that it makes code more portable between files. Also, once people get used to the concept of FILENAME in the code base it will, hopefully become obvious what it means. The main downside to this is using FILENAME in a header would give you whatever cpp file first included you (if only we got the compiler magic of __FILE__...). We would probably need to prohibit using FILENAME in .h files. Unfortunately, you cannot "using FILENAME;” because it will export all the values in each subsequent namespace. e.g.
>> 
>> namespace JSC {
>>    namespace Foo {
>> 	static int myVar { 0 };
>>    }
>>    using Foo;
>> }
>> 
>> namespace JSC {
>>    namespace Bar {
>> 	static int myVar { 0 };
>>    }
>>    using Bar;
>> }
>> 
>> does not compile.
>> 
>> Does anyone have other name recommendations for the macro name other than FILENAME? I went with file name since it describes what the substituted value will be the best, which is helpful for debugging. The other options I considered were FILE_LOCAL and PER_FILE.
>> 
> 
> I prefer FILE_LOCAL if I have to swallow this pill.
> 
>> Are there other issues that people think might occur with unified sources?
> 
> Bad news bears for files with unnamed namespaces. We will need to fix these up. There are likely few uses of them in the code base.
> 
> Dan
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170828/2736bf84/attachment.html>