[webkit-dev] Build File Maintenance (was Re: Please welcome GYP to the our dysfunctional build family)

[Kevin Ollivier]

Hi Jeremy,

On Jul 10, 2009, at 6:54 PM, Jeremy Orlow wrote:

> On Fri, Jul 10, 2009 at 6:26 PM, Kevin Ollivier <kevino at theolliviers.com 
> > wrote:
> Hi Jeremy,
>
> On Jul 10, 2009, at 3:03 PM, Jeremy Orlow wrote:
>
> [snip]
>
>> Your argument makes sense if WebKit is only built for one platform/ 
>> build-system.  Unfortunately it's not.  So the question is whether  
>> it's easier to maintain lots of different build files or whether  
>> it's easier to maintain one file that's perhaps a bit more abstract  
>> + the tool that interprets it.
>>
>> I agree that working directly in the project file for your platform  
>> is easier IF you're only developing for one platform.  But once you  
>> start maintaining more than one project file, I think GYP is a big  
>> win.
>>
>> While we hope that others will update our GYPI file when they add/ 
>> remove files, our build depends on it...so we'll definitely be  
>> keeping it in sync.  So I think the question then becomes whether  
>> it's easier for you to maintain your new build format, or whether  
>> it's easier to make it a target for GYP.  I honestly don't know  
>> what the answer is, but I think it's worth taking a closer look at  
>> GYP.
>
> Actually, the big question in regards to having GYP reduce overall  
> project maintenance is whether or not the other ports will adopt  
> GYP. If the answer is yes, then it would be more compelling for wx  
> to do so as well, assuming of course that someone implements a waf  
> backend so that we can. :-) If the answer is no, though, then GYP is  
> not reducing the amount of project maintenance work for any port  
> other than Chromium, in which case there will still be 6 build  
> systems (still 5 even if wx were to switch) and the problem I  
> originally posed in this thread will still be an issue. In that  
> case, the only way to really reduce the maintenance work of adding /  
> removing files would again be to adopt a script like the one I  
> suggested earlier.
>
> You're right.  The burden of updating the GYPI file is less than the  
> others, but it's still another file you need to change.  But note  
> that the maintainers of each project/platform/etc define which files  
> to exclude.  This is different from the other build systems where  
> the person updating the file needs to decide whether it should go  
> into the build or whether it should only be built with certain flags.
>
> Speaking of which, with waf / Python I've actually almost completely  
> automated the generation of the list of include dirs for my build  
> projects based on the source files, so that I'm not maintaining them  
> by hand anymore except for a few exceptions. And thinking about it,  
> I bet I can even mostly automate the list of source files too, by  
> having it grab all the .cpp files in the common dirs and special  
> subdirs like curl and wx, then having some include / exclude filters  
> to deal with a few special cases. :-) The question will be the  
> performance hit, but at least with the includes it's not even  
> noticeable, and I could always look into caching and changing only  
> when you do an svn up or svn add/remove.
>
> I suggest you take a look at GYP.  Much of what you're talking about  
> it already does.

Of course, GYP (and Bakefile, too) supports a wildcard file matching  
and exclusion mechanism, but that's not really as flexible as what I'm  
talking about. I'm talking about something that calculates and then  
traverses a list of subdirectories where even the set of  
subdirectories traversed are conditional upon variable values set  
earlier in the build. It's certainly possible what I'm asking for  
could be made to work, but in the end if I did get it going I'm pretty  
sure that I would be very unhappy with the syntax because, quite  
simply, a data format is not really designed to support recursion,  
looping and the like. It's, well, designed for storing data, which is  
why even conditionals are expressed in GYP using nested lists and  
property=value syntax.

So the bottom line for me is not really that I couldn't figure out a  
way to use GYP, as after all I've been using something like it for  
years, rather that when I need more flexibility and more control, the  
way to get that using a format like Bakefile or GYP does not feel  
natural or intuitive; it's more like trying to squeeze a square peg  
into a round hole. Even the question of "where should the change go" -  
extend GYP, throw it somewhere in build-webkit, write a build command  
that spits its output to GYP, etc. can become a difficult question. As  
for waf, the only question is whether the change should go in the  
common portion of the build system, or in a project-specific part, and  
refactoring as needs change is simple and straightforward, meaning I'm  
more likely to just do it rather than settle for 'good enough' just  
because refactoring would require extending the project generator  
tool, etc.

GYP's primary goal is to generate IDE project files, because that's  
what apparently many ports want. Since IDE projects have many of the  
same limitations, the GYP approach makes sense and is a nice fit for  
that use case. If IDE projects aren't the desired output, though, then  
the question of whether GYP is the best choice is no longer so clear  
cut.

> That's the sort of flexibility and ability to quickly experiment and  
> automate that scripts offer, and I suspect I will really miss that  
> if I switch back to something like Bakefile / GYP.
>
> Of course you're going to be able to experiment easier with  
> something you wrote yourself.

I didn't write waf myself. All I'm writing are build scripts. It's  
just that instead of GYP data files I'm writing Python code and data  
objects which I can manipulate using any of Python's built-in  
capabilities, meaning I can feel pretty comfortable that it will scale  
with my needs. I can even profile and find (and fix) bottlenecks in my  
build system and try to squeeze even more speed out of it if I want. :-)

>  That said, the generator for scons is only 780 lines of code  
> including a couple hundred of templates...so it doesn't seem like  
> it'd be _that_ hard to figure it out.

While it's probably not an major project, I think it's fair to say  
neither of us really knows how much work would be involved, and I  
couldn't really start real-world tests with wxWebKit until the GYP  
build system is completely landed in WebKit trunk anyway. In the  
meantime, wxWebKit has not been able to build for a couple weeks now  
in SVN, thanks to my current project generator tool, and I should be  
able to have waf support pretty much completed this weekend, so I'd  
really like to resolve that problem ASAP. The longer term issue of  
what the wx port will finally settle on can be determined later.

Regards,

Kevin

> I'm not necessarily saying you guys should be using GYP, but I  
> really think you should take a closer look at it.
>
> J

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090711/3e617e75/attachment.html>