[webkit-dev] Build system update

[Mark Rowe]

On 2011-03-23, at 12:17, Adam Barth wrote:

> On Wed, Mar 23, 2011 at 11:38 AM, Mark Rowe <mrowe at apple.com> wrote:
>> On 2011-03-23, at 03:33, Adam Barth wrote:
>>> On Wed, Mar 23, 2011 at 12:22 AM, Mark Rowe <mrowe at apple.com> wrote:
>>>> On 2011-03-22, at 23:50, Adam Barth wrote:
>>>>> On Tue, Mar 22, 2011 at 8:14 PM, Mark Rowe <mrowe at apple.com> wrote:
>>>>>> Product names for targets are redundantly declared in the Xcode project when they're already defined in the .xcconfig files.
>>>>> 
>>>>> My plan for this issue is to remove the product names from the
>>>>> xcconfig files once they're not needed by the generated xcodeproj
>>>>> files.
>>>> 
>>>> That seems backwards.  JavaScriptCore.xcconfig defines how to create JavaScriptCore.framework.  The name of the framework (e.g., PRODUCT_NAME) is an important part of that, much like the install path, Info.plist file, etc.  I don't see how we benefit from that information being spread around.
>>> 
>>> The information about how to create JavaScriptCore.framework is
>>> already spread around to a certain extent.  For example,
>>> JavaScriptCore.xcconfig does not contain the list of files that need
>>> to be built or information about the dependencies.  I'm having a hard
>>> time seeing what the practical difference is between specifying the
>>> product name in the xcodeproj file or the xcconfig file.
>> 
>> What's the argument for changing where it is specified?  Is it simply because gyp doesn't know better?
> 
> Yes.  Also, the names of the other targets are stored in the xcodeproj
> files.  I don't see this as a blocking issue, do you?
> 
>>>>>>> One area that I could use some help from one or more Apple folks is
>>>>>>> making sure that this build system integrates properly with Apple's
>>>>>>> internal build system.  Based on my (incomplete!) understanding of
>>>>>>> Apple's internal build system, there are at least two options:
>>>>>>> 
>>>>>>> 1) Apple's internal build system consumes Source in its entirety and
>>>>>>> builds a master WebKit.xcodeproj (or a Makefile), which abstracts
>>>>>>> further details about the build, such as how subsequent xcodeproj are
>>>>>>> created or how many libraries WebKit is factored into (e.g., letting
>>>>>>> us extract WTF from JavaScriptCore).
>>>>>> 
>>>>>> Making that particular change is completely unrelated to GYP.
>>>>> 
>>>>> My understanding is that it would solve the Apple internal build
>>>>> system integration issue because the Apple internal build system would
>>>>> transfer control to the master WebKit.xcodeproj, which could then
>>>>> invoke GYP to generate the remaining xcodeproj files.
>>>> 
>>>> I'm not particularly keen on a solution that involves manually invoking xcodebuild to build projects.  It becomes very complicated to ensure that the correct settings, including any overridden settings, make it down to the nested invocations of xcodebuild.
>>> 
>>> I'm not sure I've correctly communicated how I envision this approach
>>> working.  If approach (2) doesn't work out for whatever reason, I can
>>> build a mockup of how this would work, which will be more concrete.
>> 
>> You'll need to provide more detail then because I cannot see how this would work in a manner that doesn't cause the problems I have noted.
> 
> As I wrote above, we can discuss this more at a later date, if necessary.
> 
>>>>>>> 2) For each submission to Apple's internal build system, we create a
>>>>>>> branch, generate xcodeproj files, and check them into the branch.
>>>>>>> Apple's internal build system can then "svn export" the branch and see
>>>>>>> xcodeproj files, as today.
>>>>>> 
>>>>>> While this is certainly technically feasible, it would add a huge amount of overhead to the process of performing a submission.
>>>>> 
>>>>> How often do you submit WebKit to the Apple internal build system?  If
>>>>> that's sensitive information, I'm just looking for an order of
>>>>> magnitude (e.g., every revision, every hour, every day, every week).
>>>>> As a reference, the Chromium project creates one branch per day for
>>>>> daily builds.
>>>> 
>>>> I don't see how the frequency is relevant.
>>> 
>>> The relevancy arises from the observation that the overhead is
>>> proportional to the frequency.  If we only submit WebKit to Apple's
>>> internal build system once a week, then this approach won't cause too
>>> much branch proliferation.  If, on the other hand, we're submitting
>>> every revision, then we're talking about doubling or tripling the
>>> revision burn rate, which might not be desirable.
>> 
>> The rate at which we use SVN revisions or the number of branches isn't of particular concern to me.  The simple fact is that your proposal turns a single, trivial operation (create a tag) in to a sequence of more complicated operations (create a branch, check it out, run a script to generate a bunch of files, test that it builds, commit the new files, create a tag).  What used to be a single command-line operation that took a second to run is now a series of operations that takes 5+ minutes to complete.
> 
> 5 minutes doesn't sound like much of a burden given that it's paid
> only by one person (as opposed to every contributor) can easily be
> automated.

Of course it's ok with you; you're not the one that you're proposing making things 300x slower for.

>>>>> In any case, I'm glad we've found a technically feasible solution.
>>>> 
>>>> We've had at least one technically feasible solution from day zip: check in the generated project files.
>>> 
>>> From my perspective, approach (2) is more desirable than checking in
>>> generated project files because approach (2) encapsulates
>>> Apple-internal build process to Apple folks, more specifically to the
>>> Apple folks who interact with the Apple-internal build system.
>>> Checking in generated project files, on the other hand, imposes a
>>> maintenance burden on all WebKit contributors.
>> 
>> The maintenance burden you mention is minuscule in comparison to the existing requirement of updating umpteen different build systems, including the impossible-to-hand-edit Xcode project files.  This is particularly true when we're still going to be updating umpteen minus one build systems for the foreseeable future.  If the Xcode project files were the only remaining project files then I'd certainly take this argument more seriously.
> 
> Yes, unifying all our various build systems is a long journey and this
> is just the first step.

I'm glad that we agree that checking in the generated project files would create only a minuscule burden.

>> One issue that's been touched on but not stated explicitly is what the workflow will be for the average developer that wants to build WebKit.  Currently to update and build I do something like the following:
>> git svn rebase (or svn up)
>> make debug
>> 
>> Presumably people that use this workflow would need to add an extra step between those two commands to generate project files?
> 
> That would flow should work fine.  There's no need to alter it.

It would obviously work fine if the project files were checked in.  It obviously doesn't work as-is with gyp, which is what led me to raise the issue.

- Mark