[webkit-dev] Build system update

[Mark Rowe]

On 2011-03-22, at 23:50, Adam Barth wrote:

> On Tue, Mar 22, 2011 at 8:14 PM, Mark Rowe <mrowe at apple.com> wrote:
>> On 2011-03-22, at 19:16, Adam Barth wrote:
>>> WebKit-folk,
>>> 
>>> With a bunch of help from Dimitri and Eric, we now have a functioning
>>> GYP-based build for the Apple Mac port.  There are still a couple bugs
>>> we need to fix before this build system is ready for production
>>> (they're filed as blocking Bug 55018, if you're curious).  If you'd
>>> like to try out the GYP-based build, you can invoke it (on a Mac) as
>>> follows:
>>> 
>>> ./Tools/Scripts/build-webkit --gyp
>> 
>> Is this expected to work?  Running the command quoted above results in a build failure in JavaScriptCore for me (jsc is unable to find UString.h).
> 
> I'm not sure how to test with a case-sensitive file system on Mac.  (I
> didn't even know that was possible.)  How can I test that case?

Short of formatting a volume as case-sensitive HFS+, you can use Disk Utility to create a read/write disk image containing case-sensitive HFS+ and then perform a build on that disk image.

>>> If you just want to look at the generated project files, you can
>>> generate them using this command:
>>> 
>>> ./Tools/Scripts/generate-project-files
>>> 
>>> That script will create
>>> Source/JavaScriptCore/gyp/JavaScriptCore.xcodeproj and
>>> Source/WebCore/gyp/WebCore.xcodeproj (as well as a JavaScriptGlue
>>> project file, which you should feel free to ignore).
>> 
>> Is it useful for me to file bugs about these generated projects?
> 
> Definitely.  I appreciate your feedback.  If you can file them as
> blocking Bug 55018, that will ensure that they don't get lost.  I can
> check to make sure all the webkit.org requirements are satisfied, but
> I'm unable to verify that all the Apple-internal requirements are
> satisfied because they are opaque to me.

None of the issues I mentioned about the project file are related to any specific constraints that our build system places on us.  My primary concern here is ensuring that the Mac build system is understandable, repeatable, and easy to debug.  While some of the issues I've raised may seem picky, they're all things that contribute to that goal.


>> A few issues I noticed while skimming the Xcode project files…
>> The source is all hidden two levels deep in the hierarchy (inside a group named ".."?).
>> There's an oddly-named BUILT_PRODUCTS_DIR group.
>> There's a mysterious ForwardingHeaders group.
>> The .xcconfig files are in the Source group.
>> The script build phases are unnecessarily verbose ('Action "…"' or 'Postbuild "…"' where only '…' is meaningful).
> 
> Yeah, these things are not very aesthetic.  They've been on my list to
> fix, but I haven't considered them to be a blocking issue to date.  Do
> you consider these blocking issues?  I'm happy to fix them, but I'd
> like to prioritize blocking issues.

These are the types of issues that make life confusing for anyone that wants to use the Xcode project for editing, debugging, or to simply get a feel for the project.  They certainly have no impact on the actual building of the code.

>> Product names for targets are redundantly declared in the Xcode project when they're already defined in the .xcconfig files.
> 
> My plan for this issue is to remove the product names from the
> xcconfig files once they're not needed by the generated xcodeproj
> files.

That seems backwards.  JavaScriptCore.xcconfig defines how to create JavaScriptCore.framework.  The name of the framework (e.g., PRODUCT_NAME) is an important part of that, much like the install path, Info.plist file, etc.  I don't see how we benefit from that information being spread around.

>> There are mysterious extra settings in the Xcode project that serve no apparent purpose (EXECUTABLE_PREFIX, INTERMEDIATE_DIR, SHARED_INTERMEDIATE_DIR, WRAPPER_PREFIX, ALWAYS_SEARCH_USER_PATHS?)
> 
> Are these harmful?  We could figure out how to stop GYP from
> generating them, but I'm not aware of any problem they cause.

They're only harmful in the sense that they make it harder to understand the configuration of the Xcode project, which makes it harder to track down build-related issues.

>> The default configuration is Debug rather than Production.
> 
> Thanks.  I thought I had fixed that, but I'll investigate this issue
> some more.  I understand that this is a hard requirement.
> 
>> Updating Info.plist is done via a separate aggregate target rather than simply being a script phase.
> 
> Does that cause a problem?  The two seem equivalent.

There are plenty of roundabout ways in which one can achieve the same result.  That doesn't make them equally good.

>>> The main advantage of this build system over the existing Apple Mac
>>> build system is that the list of files is shared between this build
>>> system and the Chromium build system.  That means one fewer location
>>> to update when adding, removing, or renaming a file.  Over time, we
>>> should be able to share more between the build systems (because they
>>> are written in a common language) and to share the list of files
>>> between more ports.
>>> 
>>> One area that I could use some help from one or more Apple folks is
>>> making sure that this build system integrates properly with Apple's
>>> internal build system.  Based on my (incomplete!) understanding of
>>> Apple's internal build system, there are at least two options:
>>> 
>>> 1) Apple's internal build system consumes Source in its entirety and
>>> builds a master WebKit.xcodeproj (or a Makefile), which abstracts
>>> further details about the build, such as how subsequent xcodeproj are
>>> created or how many libraries WebKit is factored into (e.g., letting
>>> us extract WTF from JavaScriptCore).
>> 
>> Making that particular change is completely unrelated to GYP.
> 
> My understanding is that it would solve the Apple internal build
> system integration issue because the Apple internal build system would
> transfer control to the master WebKit.xcodeproj, which could then
> invoke GYP to generate the remaining xcodeproj files.

I'm not particularly keen on a solution that involves manually invoking xcodebuild to build projects.  It becomes very complicated to ensure that the correct settings, including any overridden settings, make it down to the nested invocations of xcodebuild.

>>> 2) For each submission to Apple's internal build system, we create a
>>> branch, generate xcodeproj files, and check them into the branch.
>>> Apple's internal build system can then "svn export" the branch and see
>>> xcodeproj files, as today.
>> 
>> While this is certainly technically feasible, it would add a huge amount of overhead to the process of performing a submission.
> 
> How often do you submit WebKit to the Apple internal build system?  If
> that's sensitive information, I'm just looking for an order of
> magnitude (e.g., every revision, every hour, every day, every week).
> As a reference, the Chromium project creates one branch per day for
> daily builds.

I don't see how the frequency is relevant.

> In any case, I'm glad we've found a technically feasible solution.


We've had at least one technically feasible solution from day zip: check in the generated project files.

- Mark