[webkit-dev] [webkit meeting notes] build systems

[Marc-Antoine Ruel]

2010/4/17 Kevin Ollivier <kevino at theolliviers.com>

> Hi Marc-Antoine,
>
> On Apr 17, 2010, at 11:26 AM, Marc-Antoine Ruel wrote:
>
> 2010/4/17 Kevin Ollivier <kevino at theolliviers.com>
>
>> Hi Maciej,
>>
>> On Apr 16, 2010, at 3:34 PM, Maciej Stachowiak wrote:
>>
>> >
>> > On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote:
>> >
>> >>
>> >> Am 16.04.2010 um 16:44 schrieb Adam Treat:
>> >>
>> >>> I am very skeptical that it is feasible to write a gyp generator that
>> would
>> >>> output QMake files.  There is a log of magic in those QMake files.  My
>> sense is
>> >>> that it would not be trivial by any means.
>> >>>
>> >>> Plus, I don't like the idea of a meta-meta generators.  Seems way to
>> mickey-
>> >>> mouse to me.
>> >>
>> >> Agreed to a certain degree. Using gyp/whatever to generate qmake files,
>> to generate Makefile/Xcode files etc seems akward to me as well.
>> >>
>> >> What we really need to resolve is adding/removing files from
>> compilation, that's the most common
>> >> task that has to be done in 5+ build systems at the moment.
>> >
>> > Besides adding, removing and renaming, the other thing that's really
>> hard is adding a new generated source rule. Although this is not needed as
>> frequently, I think anyone adding a new code generator script that has to
>> work across all WebKit ports would have a hellish time of it right now.
>> >
>> > If I had to do it myself, I would just skip any ports that don't use
>> DerivedSources.make.
>> >
>> >
>> >> So I have a new proposal:
>> >>
>> >> 1) Maintain a list of headers/source files to be compiled for ALL
>> platforms (ie. dom/Node.cpp, etc..)
>> >>
>> >> 2) Keep all existing Makefile.am, WebCore.pro etc files as "templates",
>> ie. WebCore.pro.template, with a special
>> >>   variable somewhere marking the $$HEADER_LIST$$ and the
>> $$SOURCE_LIST$$
>> >>
>> >> 3) Use a script that generates individual build files (eg. WebCore.pro)
>> from WebCore.pro.template, it only
>> >>   needs to insert the file list with the correct syntax in the correct
>> places
>> >>
>> >> 4) Keep all platform specific files to be compiled in the individual
>> build system files (eg. WebCore.pro.template)
>> >>
>> >> I think we'll never find a consensus on a single build system, there
>> are too many different needs. I only care
>> >> about the most repetitive work in order to keep the build system
>> up2date: adding/removing cross-platform files.
>> >
>> > I think the proposal above does not handle the derived sources problem.
>> It also doesn't handle files that are shared between multiple ports but not
>> all ports. It also doesn't provide project files that are directly usable by
>> IDEs, on platforms where that is the standard way to do development.
>>
>> Converting, say, a JSON list of files to some XML-based output format
>> would not be difficult at all (and I
>
>
> Like this?
> http://trac.webkit.org/browser/trunk/WebCore/WebCore.gypi
>
> - It is currently not JSON compliant. It's in fact a python file but that
> can be fixed: s/'/"/g and removing the extra commas *should* be
> sufficient.
> - It is currently chromium specific, that's what I meant by
> "de-chromification" of the gyp files. It's mainly adding more stuff and
> fixing the regexp if I'm not mistaken. I don't mind if it doesn't become the
> golden file, the goal is just to hopefully reduce the number of build
> system, nothing more.
>
>
> Yes, precisely why I mentioned JSON and later gyp, though I don't know if
> Chromium-specific is the right word here. It even has wx port files in it,
> which I don't think are built by Chromium. :) I suppose you somehow filter
> out which ones Chromium should build after the fact? If so, where does that
> logic reside?
>

mac, qt, gtk, wx, symbian and even some JSC files are listed. I'm 100% sure
the lists aren't complete and I'll need to fix them along the way,
especially JSC stuff which *is* a show stopper.

.gypi files are header files. They can be imported by other .gypi or a .gyp
file. Each .gyp file will generate a .sln, .xcodeproj, Makefile or
SConstruct, depending on the selected build system. Each "target" inside a
.gyp will generate a .vcproj or .scons.

The regex logic lives in inside "sources" entries inside a target. You can
have a list of file names, 'include' or 'exclude' with an array of regexp.
Please take a look at the very bottom of this file for an example:
http://trac.webkit.org/browser/trunk/WebCore/WebCore.gyp/WebCore.gyp

The main gyp caveat is some of its shortcut notation like /, ! and +++.
Please see the documentation<http://code.google.com/p/gyp/wiki/InputFormatReference>for
more info if interested.

WebCore.gyp already brings a lot of conditions which can make the whole
thing messy as the number of platforms increases but these can be moved off
individual .gypi files to keep the whole thing bearable. In the end, it may
even not make sense to share the gyp files accross platform if they don't
share enough build logic, it's no big deal as long as people can add and
remove files easily.

BTW, I don't why
WebCore.gypi<http://trac.webkit.org/browser/trunk/WebCore/WebCore.gypi>lists
the idl, svg idl, source files, webinspector and images as individual
lists since they all have clearly simple regexp to split them off. It still
makes sense as a logical separation though, I just want to raise the point
that it could be entirely doable to just have a one whole file list.



> Anyway, the fact that this file is actually Python (I had forgotten the
> format was JSON-like rather than straight JSON) makes things even better, as
> we only really need to handle export now, and not parse some import file
> list. i.e. for WebKitTools/Scripts/update-sources-list.py,
> getWebCoreFilesDict() basically becomes a very small script that execfile's
> the gypi file, then we run whatever filtering mechanism on it (waf has a
> list of ports we could use to do the filtering that I could probably move
> into WebKitTools/Scripts, if we don't have a pre-made Chromium solution
> here), and then passes the final source list along to
> generateWebCoreSourcesGTKandQT to generate the sources / includes for those
> platforms, and actually this solution should work for Android.mk and
> possibly jam too, as their syntax looks largely similar. Then we'd add some
> XML parsing code to grab the node for common file groups and update them for
> the MSVC projects, and then I think that mostly leaves XCode, which I think
> would be pretty similar in nature.
>

For MSVC and XCode, it makes more sense to use the native gyp support since
it makes really clean projects. FYI, VS2010 is not supported for now.

Beside that point, I agree with the rest. As you said, for other platforms,
"whatever is preferred by the committers" which may mean "don't use gyp at
all" or "use WebCore.gypi with custom scripts, not gyp".



> As long as people are willing to test out this solution with their build
> systems and help with any debugging, I would be willing to help out in
> hacking it together, though I can't promise anything in the way of time
> since this is not an immediate concern for me personally.
>

Thanks, I'll ping you later as questions show up.

---

Since this email sounds like "gyp solve world hunger", here's a short irc
log:

13:08 < maruel> manyoso: if you want to try out cmake first before I dive
into gyp stuff, feel free as I don't plan to do it in the next two weeks
13:18 < manyoso> maruel: ok, i'm waiting on othermaciej to tell me whether
apple has cmake installed in build farm
13:18 < manyoso> maruel: is potentially a show stopper apparently
13:20 < maruel> manyoso: to be clear, I really don't mind whatever is used,
as long as it can improve the productivity of my team (and by
                extension the whole community)
13:20 < manyoso> maruel: understood. and fwiw I am of the same mindset
13:21 < manyoso> maruel: if cmake turns out to have real showstoppers i am
considering helping with gyp port of apple's windows port
13:21 < manyoso> as i feel that build system proliferation is a real issue
for the whole project
13:22 < manyoso> as well as proliferation and ongoing existence of platform
specific tests
13:22 < manyoso> as the project continues to scale those two are going to
bite us in the ass more and more
13:23 < maruel> yep
13:42 < manyoso> we have ~16000 platform specific expected.txt files in
LayoutTests right now


-- 
M-A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100418/d80f6bc0/attachment.html>