[webkit-dev] [webkit meeting notes] build systems

Wed Apr 21 08:35:54 PDT 2010

Hi Marc-Antoine,

On Apr 18, 2010, at 10:47 AM, Marc-Antoine Ruel wrote:

> 2010/4/17 Kevin Ollivier <kevino at theolliviers.com>
> Hi Marc-Antoine,
> 
> On Apr 17, 2010, at 11:26 AM, Marc-Antoine Ruel wrote:
> 
>> 2010/4/17 Kevin Ollivier <kevino at theolliviers.com>
>> Hi Maciej,
>> 
>> On Apr 16, 2010, at 3:34 PM, Maciej Stachowiak wrote:
>> 
>> >
>> > On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote:
>> >
>> >>
>> >> Am 16.04.2010 um 16:44 schrieb Adam Treat:
>> >>
>> >>> I am very skeptical that it is feasible to write a gyp generator that would
>> >>> output QMake files.  There is a log of magic in those QMake files.  My sense is
>> >>> that it would not be trivial by any means.
>> >>>
>> >>> Plus, I don't like the idea of a meta-meta generators.  Seems way to mickey-
>> >>> mouse to me.
>> >>
>> >> Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well.
>> >>
>> >> What we really need to resolve is adding/removing files from compilation, that's the most common
>> >> task that has to be done in 5+ build systems at the moment.
>> >
>> > Besides adding, removing and renaming, the other thing that's really hard is adding a new generated source rule. Although this is not needed as frequently, I think anyone adding a new code generator script that has to work across all WebKit ports would have a hellish time of it right now.
>> >
>> > If I had to do it myself, I would just skip any ports that don't use DerivedSources.make.
>> >
>> >
>> >> So I have a new proposal:
>> >>
>> >> 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..)
>> >>
>> >> 2) Keep all existing Makefile.am, WebCore.pro etc files as "templates", ie. WebCore.pro.template, with a special
>> >>   variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$
>> >>
>> >> 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only
>> >>   needs to insert the file list with the correct syntax in the correct places
>> >>
>> >> 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template)
>> >>
>> >> I think we'll never find a consensus on a single build system, there are too many different needs. I only care
>> >> about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files.
>> >
>> > I think the proposal above does not handle the derived sources problem. It also doesn't handle files that are shared between multiple ports but not all ports. It also doesn't provide project files that are directly usable by IDEs, on platforms where that is the standard way to do development.
>> 
>> Converting, say, a JSON list of files to some XML-based output format would not be difficult at all (and I
>> 
>> Like this?
>> http://trac.webkit.org/browser/trunk/WebCore/WebCore.gypi
>> 
>> - It is currently not JSON compliant. It's in fact a python file but that can be fixed: s/'/"/g and removing the extra commas should be sufficient.
>> - It is currently chromium specific, that's what I meant by "de-chromification" of the gyp files. It's mainly adding more stuff and fixing the regexp if I'm not mistaken. I don't mind if it doesn't become the golden file, the goal is just to hopefully reduce the number of build system, nothing more.
> 
> Yes, precisely why I mentioned JSON and later gyp, though I don't know if Chromium-specific is the right word here. It even has wx port files in it, which I don't think are built by Chromium. :) I suppose you somehow filter out which ones Chromium should build after the fact? If so, where does that logic reside?
> 
> mac, qt, gtk, wx, symbian and even some JSC files are listed. I'm 100% sure the lists aren't complete and I'll need to fix them along the way, especially JSC stuff which is a show stopper.
> 
> .gypi files are header files. They can be imported by other .gypi or a .gyp file. Each .gyp file will generate a .sln, .xcodeproj, Makefile or SConstruct, depending on the selected build system. Each "target" inside a .gyp will generate a .vcproj or .scons.
> 
> The regex logic lives in inside "sources" entries inside a target. You can have a list of file names, 'include' or 'exclude' with an array of regexp. Please take a look at the very bottom of this file for an example:
> http://trac.webkit.org/browser/trunk/WebCore/WebCore.gyp/WebCore.gyp

May I ask why it is being done this way, that is, having one really large list of build files and using include / exclude regexs to trim the list? i.e. why is there not a WebCoreCommon.gypi, WebCoreWinCommon.gypi, WebCoreChromium.gypi, etc.? Are you guys using scripts to create WebCore.gypi?

> The main gyp caveat is some of its shortcut notation like /, ! and +++. Please see the documentation for more info if interested.
> 
> WebCore.gyp already brings a lot of conditions which can make the whole thing messy as the number of platforms increases but these can be moved off individual .gypi files to keep the whole thing bearable. In the end, it may even not make sense to share the gyp files accross platform if they don't share enough build logic, it's no big deal as long as people can add and remove files easily.
> 
> BTW, I don't why WebCore.gypi lists the idl, svg idl, source files, webinspector and images as individual lists since they all have clearly simple regexp to split them off. It still makes sense as a logical separation though, I just want to raise the point that it could be entirely doable to just have a one whole file list.
> 
>  
> Anyway, the fact that this file is actually Python (I had forgotten the format was JSON-like rather than straight JSON) makes things even better, as we only really need to handle export now, and not parse some import file list. i.e. for WebKitTools/Scripts/update-sources-list.py, getWebCoreFilesDict() basically becomes a very small script that execfile's the gypi file, then we run whatever filtering mechanism on it (waf has a list of ports we could use to do the filtering that I could probably move into WebKitTools/Scripts, if we don't have a pre-made Chromium solution here), and then passes the final source list along to generateWebCoreSourcesGTKandQT to generate the sources / includes for those platforms, and actually this solution should work for Android.mk and possibly jam too, as their syntax looks largely similar. Then we'd add some XML parsing code to grab the node for common file groups and update them for the MSVC projects, and then I think that mostly leaves XCode, which I think would be pretty similar in nature. 
> 
> For MSVC and XCode, it makes more sense to use the native gyp support since it makes really clean projects. FYI, VS2010 is not supported for now.

Yes, if all MSVC and XCode projects are planning to switch over anyway, it's probably not worth bothering with. Though if some choose not to, we could actually re-use the chunk of Gyp code that creates the file lists and just inject that into the project file.

> Beside that point, I agree with the rest. As you said, for other platforms, "whatever is preferred by the committers" which may mean "don't use gyp at all" or "use WebCore.gypi with custom scripts, not gyp".
> 
>  
> As long as people are willing to test out this solution with their build systems and help with any debugging, I would be willing to help out in hacking it together, though I can't promise anything in the way of time since this is not an immediate concern for me personally.
> 
> Thanks, I'll ping you later as questions show up.

Sure. :)

Thanks,

Kevin

> ---
> 
> Since this email sounds like "gyp solve world hunger", here's a short irc log:
> 
> 13:08 < maruel> manyoso: if you want to try out cmake first before I dive into gyp stuff, feel free as I don't plan to do it in the next two weeks
> 13:18 < manyoso> maruel: ok, i'm waiting on othermaciej to tell me whether apple has cmake installed in build farm
> 13:18 < manyoso> maruel: is potentially a show stopper apparently
> 13:20 < maruel> manyoso: to be clear, I really don't mind whatever is used, as long as it can improve the productivity of my team (and by
>                 extension the whole community)
> 13:20 < manyoso> maruel: understood. and fwiw I am of the same mindset
> 13:21 < manyoso> maruel: if cmake turns out to have real showstoppers i am considering helping with gyp port of apple's windows port
> 13:21 < manyoso> as i feel that build system proliferation is a real issue for the whole project
> 13:22 < manyoso> as well as proliferation and ongoing existence of platform specific tests
> 13:22 < manyoso> as the project continues to scale those two are going to bite us in the ass more and more
> 13:23 < maruel> yep
> 13:42 < manyoso> we have ~16000 platform specific expected.txt files in LayoutTests right now
> 
> 
> -- 
> M-A

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100421/fd311f25/attachment.html>