[webkit-dev] Common build system (was Re: WebKit Wishes)

[Mark Rowe]

On 2013-01-30, at 17:14, Dirk Pranke <dpranke at chromium.org> wrote:

> On Wed, Jan 30, 2013 at 4:15 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>> 
>> On Jan 30, 2013, at 3:24 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>> 
>>> On Wed, Jan 30, 2013 at 1:50 PM, Filip Pizlo <fpizlo at apple.com> wrote:
>>>> Thanks for sharing this.
>>>> 
>>>> On Jan 30, 2013, at 1:28 PM, Eric Seidel <eric at webkit.org> wrote:
>>>> 
>>>> I wish we only had one build system (it were easy to add/remove/move files).
>>>> 
>>>> I believe changes like http://trac.webkit.org/changeset/74849 are an
>>>> unhealthy sign for the project.  Adam is not the only person who has chosen
>>>> to empty files instead of removing them.  The pain of updating 8 build
>>>> system is so great, we jump through hoops to avoid it.  Which means it took
>>>> us months to move JavaScriptCore/wtf to WTF, and will take us years to kill
>>>> WebCore/platform.
>>>> 
>>>> 
>>>> +1
>>>> 
>>>> This is a hard problem.  It is a problem worth solving.
>>>> 
>>>> Do you have more thoughts on this, particularly since you know quite well
>>>> how both Xcode and gyp work?
>>>> 
>>>> I suspect this is one of those things that it would be hard to achieve
>>>> consensus on since there are so many stakeholders.  But it may be fruitful
>>>> to have a "what if" discussion about what this might look like.
>>>> 
>>> 
>>> I think we can solve this problem if we agree that we want to. I think
>>> we haven't in the past mostly because we couldn't reach a consensus
>>> that it was worth solving enough to really try.
>>> 
>>> I would love to see this fixed and would be glad to work on it. I
>>> think we should at least pursue this far enough to fully understand
>>> what our options are and what the costs and tradeoffs might be; does
>>> anyone disagree, and is anyone else willing to help pitch in?
>>> 
>>> I think there are several possible ways we could solve this. One would
>>> be to switch to a common meta-build system. My understanding is that
>>> Apple's internal production build processes impose certain constraints
>>> here that I don't fully understand, but I know we've discussed the
>>> possibility of checking in generated project files as a workaround.
>>> Maybe there are other options as well to those constraints? I would
>>> love to discuss this further w/ someone from Apple ...
>> 
>> It's far simplest for us if:
>> (a) There is an Xcode project (or a Makefile) that builds the Mac port checked in to source control.
>> (b) The generated project invokes only tools that are part of the default Mac OS X install.
>> 
>> It may not be completely impossible to violate these requirements but it will require a lot of bureaucracy.
>> 
>>> (Also, just to get this out of the way, I don't think gyp needs to be
>>> the solution).
>>> 
>>> Another alternative would be to write a script that did support at
>>> least the common use cases (add/move/delete files). There have been
>>> attempts in the past, but they have foundered on at least some
>>> perceived skepticism over how well this would work w/ XCode. That
>>> said, I don't think we've really pushed this to see. At some point
>>> this script might turn into a meta-meta-build system, which might be
>>> silly but also be the shortest path to the finish line.
>>> 
>>> I suggest if there is interest in this we can start a new thread to
>>> discuss further ...
>> 
>> My preference would be to use a common meta-build system with a comfortably human-readable and human-editable syntax, and checked in generated project files for those ports that need them.
>> 
>> I think a key to making this work is to get Chromium and the Apple Mac port onto a common build system, which will probably require both Google and Apple ponying up at least one person to work on this project for a reasonable period of time.
>> 
>> I think the plausible meta-build-systems to use would be CMake and Gyp, but in both cases it may be necessary to modify them to work well for everyone.
>> 
> 
> Premake might also be an option, though I wouldn't necessarily vote
> for it. Gyp's syntax is ... awkward ... but apart from that might just
> work for checking in generated apple mac xcode projects. We should try
> it (shouldn't be too hard, since abarth had it working at one point).

I think Eric and/or Adam had JavaScriptCore building with gyp at one point. I’m not sure if they ever got to the other projects.

> I would consider changing or improving gyp's syntax to be on the table
> if it was needed to reach the goal.

For what it’s worth, I also find the gyp syntax to be unpleasant. It feels as though it was optimized for being processed by a machine rather than for being written and maintained by humans.

> CMake was originally considered a non-starter for chromium since the
> XCode and VS projects it produced didn't look or feel like native
> projects. However, we've since switched to ninja for most things so
> I'm less sure how important this is now. I've never been a huge fan of
> the CMake syntax but I haven't used it enough to really have an
> informed opinion.

Generating well-structured Xcode projects is still something that is important for any meta build system that we would consider adopting for the Mac. It would be substantially more challenging for us to support an alternative system for building our production builds. If there are substantial advantages to using ninja for engineering builds then we may wish to support both to get the best behavior in each environment.

The non-natively structured Xcode projects is one thing that turned me off CMake when I looked at it in the past. I’m not sure if this has improved since then. If not, then I don’t think CMake is worth spending much time on.

> Regarding "(b) The generated project invokes only tools that are part
> of the default Mac OS X install": invoking tools that are checked into
> the WK repo is also possible, right, since we invoke scripts now? So,
> hypothetically, could we check in a copy of the ninja binary and build
> with that?

Checking in binaries isn’t an option for us, and isn’t a particularly scalable approach anyway given the number of platforms that WebKit can build on. If we require an external tool as a dependency to build WebKit from source then we’d need to check in the source for the tool and build it prior to building WebKit proper. This obviously introduces more complexity so it would be preferable to keep the dependencies to a minimum.

- Mark