[webkit-dev] Growing tired of long build times? Check out this awesome new way to speed up your build... soon (HINT: It's not buying a new computer)

[Keith Miller]

Hello fellow Webkittens,

Are you growing tired of long cat naps while waiting 45 minutes for WebKit to build? (I’ve definitely never done this 🤐!)
Do you want WebKit to build up to 3-4x faster on a clean build?*
Does seeing your incremental build… stay the same sound good?

You’ll be purring with excitement after you switch to unified source builds!**

* Building JSC with CMake, including CMake configuration time, went from ~8m15s to ~2m30s on my MBP using unified sources only on cpp files in JavaScriptCore. Final results on WebKit may vary. Using unified sources may cause sudden feelings of euphoria and productivity (or grumpiness from lack of naps…). Not responsible for any bugs in extra patches produced by using unified source builds (excluding build system related bugs) except where required by law (pizlo at apple.com <mailto:pizlo at apple.com>).

** Soon. Unified source builds haven’t been landed yet.

How are things going to change? Great question! Let’s find out!

When you fire off the build script it will automagically paw your source files into bundles. The size of each bundle is speculatively going to be 8 .cpp files but that’s up for experimentation. From my testing so far, the compile time for a bundle is at most 20% (~6s vs ~7s for the Air directory) longer than the longest file it includes. Given that most of the build time in incremental builds is scanning dependencies, this change is probably only barely noticeable.

Bundling happens as part of the CMake configuration process, in the CMake build. In the Xcode build it’s more complicated since we cannot dynamically choose the files we are going to compile. The only solution I know of is to pre-list a bunch of files for the build to dump files into. Occasionally, we’ll need to add more files to the build list.

When you add or remove a file in the project, the cost is higher. The current plan to bundle source files is by directory. This means that when a .cpp file is added or removed you will rebalance the bundles in its directory, and you will have to rebuild up to the full directory that file is in. 

My goal is to make all of this invisible to developers as much as possible. I believe, for the most part, things should operate mostly as they have before. As the details on unified source builds get finalized, I’m sending out a separate email documenting any workflow changes. I’ll also do my best to answer any questions or concerns.

“But Keith, am I going to have to make major changes to the way I develop WebKit?” Yes and, hopefully, no. There is one major common workflow change that comes with unified sources, the naming of static variables and functions. Since C++ does not have a way to only declare a static function or variable in some restricted top level scope we are going to need to create some work around. The choice I’m thinking we’ll go with is to have an auto-generated macro value FILENAME, which is similar to __FILE__ except that it doesn’t have file extension (We can’t use FILE since that’s actually a type in C 🙁). This macro would be redefined in the unified source directly above the file that is about to be included. Then in each cpp file that has static functions/variables developers would have:

namespace FILENAME {
	static int myVariable { 0 };
	static int myFunction() { return myVariable; }
}

The advantage of using a macro over textually writing the filename is that it makes code more portable between files. Also, once people get used to the concept of FILENAME in the code base it will, hopefully become obvious what it means. The main downside to this is using FILENAME in a header would give you whatever cpp file first included you (if only we got the compiler magic of __FILE__...). We would probably need to prohibit using FILENAME in .h files. Unfortunately, you cannot "using FILENAME;” because it will export all the values in each subsequent namespace. e.g.

namespace JSC {
    namespace Foo {
	static int myVar { 0 };
    }
    using Foo;
}

namespace JSC {
    namespace Bar {
	static int myVar { 0 };
    }
    using Bar;
}

does not compile.

Does anyone have other name recommendations for the macro name other than FILENAME? I went with file name since it describes what the substituted value will be the best, which is helpful for debugging. The other options I considered were FILE_LOCAL and PER_FILE.

Are there other issues that people think might occur with unified sources? Note, It’s also likely that there are some source files that need to be excluded from the unified source builds for various reasons.

“Is there anything else I need to do with unified sources?”, Yes! Enjoy the faster builds!

To help visualize how much of a difference unified sources can make for you, I’ve illustrated this helpful graph on the WebKit developer experience:



* Due to size constraints this graph is not to scale.


Cheers,
Keith

P.S.

For those interested there was a long process in deciding how to improve builds:

The first thing I looked at was using C++ modules.

There were a couple of problems with C++ modules. First, different compliers support modules in different ways. This was not a major concern since we probably would have gone with Clang’s version of modules. As far as I know, almost all developers either use clang or could likely switch to clang for development. The bigger problem with Clang modules is that it seemed like we would have a rough early adopter problem. Additionally, since a lot of Apple’s internal SDK’s don’t seem to support modules it was likely to be a major slog to get everything working correctly. In a few years I think it will probably be worth looking at C++ modules again as, in theory, C++ modules should be better than unified sources.

Since we decided against C++ modules that really only leaves us with some form of unified sources. Obviously, doing all in one builds would be out of the question. While all in ones would have been great for clean builds, it also means incremental builds are essentially the same as clean builds. So, I concluded that we should go with some kind of automated unified build file solution. An automated unified build file solution means that most of the time the build system is opaque to developers (well as much as is possible with a unified source build).

Since the static variable naming problem was likely to be the biggest issue with unified sources, it was important to create a system that made this a pain free as possible. There were other options we could go with. We could have a script that parses .cpp files and looks for static variables with the same name and yells at you. This is pretty annoying and wouldn’t help actual development when people hit an error. For example, I know I never run the check-webkit-style script until I’m putting a patch up for review.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170828/e61e1405/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Dope chart.jpg
Type: image/jpeg
Size: 152278 bytes
Desc: not available
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170828/e61e1405/attachment-0001.jpg>