[webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)

[Kevin Ollivier]

Hi David,

On Jul 10, 2009, at 11:06 AM, David Kilzer wrote:

>
>> So, does anyone think this would be a bad idea, or have any
>> alternate suggestions on how to improve things?
>
> What about adding support for waf to gyp?

If we're talking about Chromium, waf is much faster for building  
WebKit than scons is, so yeah, I would say adding the ability to use  
it instead of scons is a no-brainer.

As for wx, while I'm open to the idea of using gyp to help others out  
or simplify matters for the core devs, for us I'm pretty sure it will  
mostly just be one more added layer to our build system. I've been  
using Bakefile for years now and while I thought the idea was great at  
first, I've found it mostly a pain to work with. Some of that is due  
to Bakefile itself, of course (like the latest bug...), but a lot of  
it is just the general approach and issues inherent with adding one  
more tool to the toolchain. Even though there's some sort of meta-tool  
generating the build projects, it only reduces, not removes, the need  
to understand and deal with the native project formats it outputs.  
Also, since the meta-tool can have bugs or missing features too,  
there's an additional point of failure added, and also added work in  
terms of maintaining the build system. Plus, any time you want to add  
a feature, you should look into how to implement it in the various  
formats that gyp supports to avoid hacks, a process which can  
sometimes require pretty creative solutions for less capable build  
tools. And there's also the fact that, if wx uses gyp, people are  
going to start asking for support for their favorite project format,  
which is just one more thing to deal with.

In short, I think I've just come to be soured on the approach in  
general. I think I've come to agree with the point that the waf site  
makes about domain-specific languages not being a good idea for build  
systems, and when you use a tool like Bakefile or gyp, you not only  
need to become familiar with that tool's project file language, but  
also to some level with the formats of the project files that it  
outputs in order to debug or add new capabilities. With waf, all I  
ever really need to know is Python, which I already know, and the waf  
APIs, which I'm learning but for the most part have been pretty simple  
and straightforward to work with. The bottom line for me is that  
anything that has me spending less time on maintaining the build  
system and more on the actual product is a good thing in my book. ;-)

Thanks,

Kevin

>> The only format I'm not sure if we could automate > reliably would  
>> be the XCode format (at least, on non-Mac
>> machines), because IIUC we'd need some sort of parser for it,
>> but Apple is the only port maintaining those directly IIUC,
>> as now Chromium will be using GYP to update their XCode
>> projects.
>
> If you hand-edit Xcode project files enough times you start to  
> understand them, but you also may go insane in the process.  I don't  
> know that you have to write a full parser for it, but there is some  
> non-trivial, minimal structure you have to understand to update the  
> file properly.
>
>> So even if we couldn't solve the XCode issue, that
>> would drop it to updating two locations tops.
>
> I count 6 build systems in use currently (SCons support was added  
> and removed within the last year):
>
> - Apple's Xcode
> - Apple's vcproj (also used by at least one other Windows port)
> - wx Bakefile (which will be replaced by waf soon)
> - Qt Qmake
> - GTK GNUMakefile
> - Google's gyp (added recently)
>
> Are any of the other ports going to switch to generating their build  
> files using gyp?
>
> Dave
>
>
>
> ----- Original Message ----
>> From: Kevin Ollivier <kevino at theolliviers.com>
>> To: Dimitri Glazkov <dglazkov at chromium.org>
>> Cc: Mark Mentovai <mark at chromium.org>; WebKit Development <webkit-dev at lists.webkit.org 
>> >
>> Sent: Friday, July 10, 2009 8:52:57 AM
>> Subject: [webkit-dev] Build File Maintenance (was Re: Please  
>> welcome GYP to the our dysfunctional build family)
>>
>> Hi Dimitri and all,
>>
>> Congrats on getting this into WebKit! Actually, I'm in the middle  
>> of a build
>> system switch as well - to waf, a re-write of scons that removed  
>> many of the
>> performance issues related to searching and calculating  
>> dependencies, and which
>> has added some nice features as well (such as .app bundle  
>> building). I haven't
>> completed the switch so I can't do preliminary benchmarks, but I'm  
>> pretty sure
>> it's actually as fast or faster than make on *nix/Mac. (And BTW, it  
>> will
>> probably make Apple devs happy to hear that I'm no longer using the  
>> horrid
>> build-wxwebkit bash script to manage the build, but instead have  
>> integrated
>> everything into build-webkit finally!)
>>
>> The main reason I bring this up, though, is because I think this  
>> sort of thing
>> shows that we're unlikely to centralize our build systems any time  
>> soon, and I
>> feel a bit sorry for the core devs who, as they accept new ports,  
>> are probably
>> finding it more and more tedious, if not difficult, to make sure  
>> all the
>> projects get updated by a change to the common parts of the build.  
>> They've been
>> very helpful in terms of trying to keep the ports in shape when  
>> they make
>> changes, and I feel like I'd like to do what I can to make it  
>> easier and faster
>> to keep the other ports in sync.
>>
>> At one point, I started on a script (located in
>> WebKitTools/Scripts/update-sources-list.py) whose idea was to take  
>> the list of
>> common sources from one file, and make changes to MSVC, Qt, GTK,  
>> etc. build
>> systems, so that WebKit devs need only add the file once, run the  
>> script, and
>> commit the results. I got it as far as theoretically generating the  
>> Qt and GTK
>> file lists, but I don't think anyone on those ports tried  
>> integrating it into
>> their build system, and I sort of moved on to other things.
>>
>> Unfortunately, right now I'm really swamped (my build system  
>> rewrite was
>> prompted by WebKit exceeding internal, hardcoded, Bakefile limits,  
>> not by
>> choice), but if a common location for the source files list could  
>> be decided
>> upon, I really think a script would be a simple matter to write up  
>> (even in Perl
>> :P ), and I think it would probably save developers time and reduce  
>> build
>> breakages as well, which I think would add up to a lot of saved  
>> time for a lot
>> of people over the long term. The only format I'm not sure if we  
>> could automate
>> reliably would be the XCode format (at least, on non-Mac machines),  
>> because IIUC
>> we'd need some sort of parser for it, but Apple is the only port  
>> maintaining
>> those directly IIUC, as now Chromium will be using GYP to update  
>> their XCode
>> projects. So even if we couldn't solve the XCode issue, that would  
>> drop it to
>> updating two locations tops.
>>
>> So, does anyone think this would be a bad idea, or have any alternate
>> suggestions on how to improve things? If not, is anyone willing to  
>> take the ball
>> and run with it? I'd be willing to invest some more time in it, but  
>> my ability
>> to commit to it over the next couple weeks at least would be very  
>> limited.
>>
>> Thanks,
>>
>> Kevin
>>
>> On Jul 9, 2009, at 9:23 PM, Dimitri Glazkov wrote:
>>
>>> Dear WebKiteurs,
>>>
>>> In our persisting quest to be more like a common WebKit port, we  
>>> have
>>> added Chromium build files to the tree this afternoon. These files  
>>> are
>>> WebCore/WebCore.gypi and JavaScriptCore/JavaScriptCore.gypi and they
>>> are the GYP include files. As you may know, we use GYP
>>> (http://code.google.com/p/gyp) for generating MSVC, XCode, Scons,  
>>> and
>>> even Make projects for Chromium.
>>>
>>> We are rather fond of GYP. Perhaps it is because it allows us to
>>> maintain one set of project files for all three Chromium platforms;
>>>
>>> or maybe because it lets us to do things like WebCore.gypi, where we
>>> can just mindlessly add all project files to the list and then use
>>> various neat GYP filtering facilities to narrow them down to sets  
>>> that
>>> are relevant for specific builds;
>>>
>>> or maybe because it easifies creating cross-platform and
>>> cross-build-system targets, actions, and rules;
>>>
>>> or maybe because we just love saying "Gyp!"
>>>
>>> I don't truthfully know.
>>>
>>> What I do know is that starting now, we'd love for you to remember
>>> WebCore.gypi and JavaScriptCore.gypi when you are adding or removing
>>> files from WebCore or JavaScriptCore. Thanks to the power of GYP,  
>>> you
>>> don't have worry whether this file will be used by Chromium: the  
>>> rule
>>> is that if there's a project file change, it applies to the *.gypi
>>> files. The format is very simple and intuitive, a simple Python/ 
>>> JSONey
>>> list+dict.
>>>
>>> Thank you for your attention, men and women of WebKit.
>>>
>>> :DG<
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>