[webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)

[Jeremy Orlow]

On Fri, Jul 10, 2009 at 1:14 PM, Kevin Ollivier <kevino at theolliviers.com>wrote:

> Hi David,
>
> On Jul 10, 2009, at 11:06 AM, David Kilzer wrote:
>
>
>>  So, does anyone think this would be a bad idea, or have any
>>> alternate suggestions on how to improve things?
>>>
>>
>> What about adding support for waf to gyp?
>>
>
> If we're talking about Chromium, waf is much faster for building WebKit
> than scons is, so yeah, I would say adding the ability to use it instead of
> scons is a no-brainer.
>
> As for wx, while I'm open to the idea of using gyp to help others out or
> simplify matters for the core devs, for us I'm pretty sure it will mostly
> just be one more added layer to our build system. I've been using Bakefile
> for years now and while I thought the idea was great at first, I've found it
> mostly a pain to work with. Some of that is due to Bakefile itself, of
> course (like the latest bug...), but a lot of it is just the general
> approach and issues inherent with adding one more tool to the toolchain.
> Even though there's some sort of meta-tool generating the build projects, it
> only reduces, not removes, the need to understand and deal with the native
> project formats it outputs. Also, since the meta-tool can have bugs or
> missing features too, there's an additional point of failure added, and also
> added work in terms of maintaining the build system. Plus, any time you want
> to add a feature, you should look into how to implement it in the various
> formats that gyp supports to avoid hacks, a process which can sometimes
> require pretty creative solutions for less capable build tools. And there's
> also the fact that, if wx uses gyp, people are going to start asking for
> support for their favorite project format, which is just one more thing to
> deal with.
>
> In short, I think I've just come to be soured on the approach in general. I
> think I've come to agree with the point that the waf site makes about
> domain-specific languages not being a good idea for build systems, and when
> you use a tool like Bakefile or gyp, you not only need to become familiar
> with that tool's project file language, but also to some level with the
> formats of the project files that it outputs in order to debug or add new
> capabilities. With waf, all I ever really need to know is Python, which I
> already know, and the waf APIs, which I'm learning but for the most part
> have been pretty simple and straightforward to work with. The bottom line
> for me is that anything that has me spending less time on maintaining the
> build system and more on the actual product is a good thing in my book. ;-)


Your argument makes sense if WebKit is only built for one
platform/build-system.  Unfortunately it's not.  So the question is whether
it's easier to maintain lots of different build files or whether it's easier
to maintain one file that's perhaps a bit more abstract + the tool that
interprets it.

I agree that working directly in the project file for your platform is
easier IF you're only developing for one platform.  But once you start
maintaining more than one project file, I think GYP is a big win.

While we hope that others will update our GYPI file when they add/remove
files, our build depends on it...so we'll definitely be keeping it in sync.
 So I think the question then becomes whether it's easier for you to
maintain your new build format, or whether it's easier to make it a target
for
GYP.  I honestly don't know what the answer is, but I think it's worth
taking a closer look at GYP.

J


>
>
> Thanks,
>
> Kevin
>
>
>  The only format I'm not sure if we could automate > reliably would be the
>>> XCode format (at least, on non-Mac
>>> machines), because IIUC we'd need some sort of parser for it,
>>> but Apple is the only port maintaining those directly IIUC,
>>> as now Chromium will be using GYP to update their XCode
>>> projects.
>>>
>>
>> If you hand-edit Xcode project files enough times you start to understand
>> them, but you also may go insane in the process.  I don't know that you have
>> to write a full parser for it, but there is some non-trivial, minimal
>> structure you have to understand to update the file properly.
>>
>>  So even if we couldn't solve the XCode issue, that
>>> would drop it to updating two locations tops.
>>>
>>
>> I count 6 build systems in use currently (SCons support was added and
>> removed within the last year):
>>
>> - Apple's Xcode
>> - Apple's vcproj (also used by at least one other Windows port)
>> - wx Bakefile (which will be replaced by waf soon)
>> - Qt Qmake
>> - GTK GNUMakefile
>> - Google's gyp (added recently)
>>
>> Are any of the other ports going to switch to generating their build files
>> using gyp?
>>
>> Dave
>>
>>
>>
>> ----- Original Message ----
>>
>>> From: Kevin Ollivier <kevino at theolliviers.com>
>>> To: Dimitri Glazkov <dglazkov at chromium.org>
>>> Cc: Mark Mentovai <mark at chromium.org>; WebKit Development <
>>> webkit-dev at lists.webkit.org>
>>> Sent: Friday, July 10, 2009 8:52:57 AM
>>> Subject: [webkit-dev] Build File Maintenance (was Re: Please welcome GYP
>>> to the our dysfunctional build family)
>>>
>>> Hi Dimitri and all,
>>>
>>> Congrats on getting this into WebKit! Actually, I'm in the middle of a
>>> build
>>> system switch as well - to waf, a re-write of scons that removed many of
>>> the
>>> performance issues related to searching and calculating dependencies, and
>>> which
>>> has added some nice features as well (such as .app bundle building). I
>>> haven't
>>> completed the switch so I can't do preliminary benchmarks, but I'm pretty
>>> sure
>>> it's actually as fast or faster than make on *nix/Mac. (And BTW, it will
>>> probably make Apple devs happy to hear that I'm no longer using the
>>> horrid
>>> build-wxwebkit bash script to manage the build, but instead have
>>> integrated
>>> everything into build-webkit finally!)
>>>
>>> The main reason I bring this up, though, is because I think this sort of
>>> thing
>>> shows that we're unlikely to centralize our build systems any time soon,
>>> and I
>>> feel a bit sorry for the core devs who, as they accept new ports, are
>>> probably
>>> finding it more and more tedious, if not difficult, to make sure all the
>>> projects get updated by a change to the common parts of the build.
>>> They've been
>>> very helpful in terms of trying to keep the ports in shape when they make
>>> changes, and I feel like I'd like to do what I can to make it easier and
>>> faster
>>> to keep the other ports in sync.
>>>
>>> At one point, I started on a script (located in
>>> WebKitTools/Scripts/update-sources-list.py) whose idea was to take the
>>> list of
>>> common sources from one file, and make changes to MSVC, Qt, GTK, etc.
>>> build
>>> systems, so that WebKit devs need only add the file once, run the script,
>>> and
>>> commit the results. I got it as far as theoretically generating the Qt
>>> and GTK
>>> file lists, but I don't think anyone on those ports tried integrating it
>>> into
>>> their build system, and I sort of moved on to other things.
>>>
>>> Unfortunately, right now I'm really swamped (my build system rewrite was
>>> prompted by WebKit exceeding internal, hardcoded, Bakefile limits, not by
>>> choice), but if a common location for the source files list could be
>>> decided
>>> upon, I really think a script would be a simple matter to write up (even
>>> in Perl
>>> :P ), and I think it would probably save developers time and reduce build
>>> breakages as well, which I think would add up to a lot of saved time for
>>> a lot
>>> of people over the long term. The only format I'm not sure if we could
>>> automate
>>> reliably would be the XCode format (at least, on non-Mac machines),
>>> because IIUC
>>> we'd need some sort of parser for it, but Apple is the only port
>>> maintaining
>>> those directly IIUC, as now Chromium will be using GYP to update their
>>> XCode
>>> projects. So even if we couldn't solve the XCode issue, that would drop
>>> it to
>>> updating two locations tops.
>>>
>>> So, does anyone think this would be a bad idea, or have any alternate
>>> suggestions on how to improve things? If not, is anyone willing to take
>>> the ball
>>> and run with it? I'd be willing to invest some more time in it, but my
>>> ability
>>> to commit to it over the next couple weeks at least would be very
>>> limited.
>>>
>>> Thanks,
>>>
>>> Kevin
>>>
>>> On Jul 9, 2009, at 9:23 PM, Dimitri Glazkov wrote:
>>>
>>>  Dear WebKiteurs,
>>>>
>>>> In our persisting quest to be more like a common WebKit port, we have
>>>> added Chromium build files to the tree this afternoon. These files are
>>>> WebCore/WebCore.gypi and JavaScriptCore/JavaScriptCore.gypi and they
>>>> are the GYP include files. As you may know, we use GYP
>>>> (http://code.google.com/p/gyp) for generating MSVC, XCode, Scons, and
>>>> even Make projects for Chromium.
>>>>
>>>> We are rather fond of GYP. Perhaps it is because it allows us to
>>>> maintain one set of project files for all three Chromium platforms;
>>>>
>>>> or maybe because it lets us to do things like WebCore.gypi, where we
>>>> can just mindlessly add all project files to the list and then use
>>>> various neat GYP filtering facilities to narrow them down to sets that
>>>> are relevant for specific builds;
>>>>
>>>> or maybe because it easifies creating cross-platform and
>>>> cross-build-system targets, actions, and rules;
>>>>
>>>> or maybe because we just love saying "Gyp!"
>>>>
>>>> I don't truthfully know.
>>>>
>>>> What I do know is that starting now, we'd love for you to remember
>>>> WebCore.gypi and JavaScriptCore.gypi when you are adding or removing
>>>> files from WebCore or JavaScriptCore. Thanks to the power of GYP, you
>>>> don't have worry whether this file will be used by Chromium: the rule
>>>> is that if there's a project file change, it applies to the *.gypi
>>>> files. The format is very simple and intuitive, a simple Python/JSONey
>>>> list+dict.
>>>>
>>>> Thank you for your attention, men and women of WebKit.
>>>>
>>>> :DG<
>>>> _______________________________________________
>>>> webkit-dev mailing list
>>>> webkit-dev at lists.webkit.org
>>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>>
>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>
>>
>>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090710/f85d5ecc/attachment.html>