[webkit-dev] [webkit meeting notes] build systems

Sat Apr 17 11:11:06 PDT 2010

Hi Maciej,

On Apr 16, 2010, at 3:34 PM, Maciej Stachowiak wrote:

> 
> On Apr 16, 2010, at 8:09 AM, Nikolas Zimmermann wrote:
> 
>> 
>> Am 16.04.2010 um 16:44 schrieb Adam Treat:
>> 
>>> I am very skeptical that it is feasible to write a gyp generator that would
>>> output QMake files.  There is a log of magic in those QMake files.  My sense is
>>> that it would not be trivial by any means.
>>> 
>>> Plus, I don't like the idea of a meta-meta generators.  Seems way to mickey-
>>> mouse to me.
>> 
>> Agreed to a certain degree. Using gyp/whatever to generate qmake files, to generate Makefile/Xcode files etc seems akward to me as well.
>> 
>> What we really need to resolve is adding/removing files from compilation, that's the most common
>> task that has to be done in 5+ build systems at the moment.
> 
> Besides adding, removing and renaming, the other thing that's really hard is adding a new generated source rule. Although this is not needed as frequently, I think anyone adding a new code generator script that has to work across all WebKit ports would have a hellish time of it right now.
> 
> If I had to do it myself, I would just skip any ports that don't use DerivedSources.make.
> 
> 
>> So I have a new proposal:
>> 
>> 1) Maintain a list of headers/source files to be compiled for ALL platforms (ie. dom/Node.cpp, etc..)
>> 
>> 2) Keep all existing Makefile.am, WebCore.pro etc files as "templates", ie. WebCore.pro.template, with a special
>>   variable somewhere marking the $$HEADER_LIST$$ and the $$SOURCE_LIST$$
>> 
>> 3) Use a script that generates individual build files (eg. WebCore.pro) from WebCore.pro.template, it only
>>   needs to insert the file list with the correct syntax in the correct places
>> 
>> 4) Keep all platform specific files to be compiled in the individual build system files (eg. WebCore.pro.template)
>> 
>> I think we'll never find a consensus on a single build system, there are too many different needs. I only care
>> about the most repetitive work in order to keep the build system up2date: adding/removing cross-platform files.
> 
> I think the proposal above does not handle the derived sources problem. It also doesn't handle files that are shared between multiple ports but not all ports. It also doesn't provide project files that are directly usable by IDEs, on platforms where that is the standard way to do development.

Converting, say, a JSON list of files to some XML-based output format would not be difficult at all (and I believe we can automatically convert the XCode project files from binary to plist and back, though IIRC there might be some UUID handling issues to consider), so we could patch the IDE files much like we patch the other ports. As for the other cases, such as additions / removals of things shared by multiple ports and the derived sources problem (that one probably would indeed need some 'template' to insert into project files), I think these could be added over time if we feel it'd bring a large benefit, but I think even just covering the common case of cross-platform source file maintenance is already a huge win for the project. I maintained Bakefile projects for years, and I'd say 80-90% of the time when a change broke my build, it was one of these common source file additions / removals. And it usually happened several times every week.

I personally think the way to look at it is to start by solving the simple stuff that could be solved quickly, as in my experience that makes it far more likely to actually get done. If, say, gyp -> Gtk / Qt / MSVC (/ XCode?) build file lists could be done in a weekend of hacking and some devs are interested in working on it, why not? 

> Once we start solving problems like that, I suspect we end up with something closer in complexity to Gyp or CMake.

True, but I think the real problem that we're not addressing in this discussion is that different ports have different sets of requirements, meaning their own evaluation process would lead them to choose different tools. If we want a 'one size fits all' build system, the first step is getting each port to come together and consolidate the requirements, and there will most likely need to be some compromises involved as some ports may have to agree to move to a tool that's not really as well suited for their project as the one they're using now. 

For example, a primary reason tools like Gyp and Bakefile exist is that for some people, lack of a 100% native IDE-based build system is a deal breaker. For others, like myself, I want low maintenance, completely cross-platform, highly automated and highly scriptable, which are actually features the IDE projects don't fare very well in. So one man's feature is another man's drawback.

There are also factors besides features that are important as well. I think it's also important to remember that from each port's perspective, one potentially big factor in build system choice is also making users comfortable with contributing. For GTK, for example, that may mean that using autotools, the build system most likely to be familiar to potential contributors, is in part a way of making contributing a little less intimidating on a big project like WebKit. Similar with qmake, XCode, etc. I think that a big part of build system decision making is based not necessarily around features, but around being in the developers' comfort zones. So my gut impression is that it's going to be very difficult to find an existing tool that solves all the issues like this for most / all ports in a way that makes everyone happy.

As the saying goes, sometimes the perfect is indeed the enemy of the good. I personally think a quick and simple solution along the lines of what Nikolas and I were talking about maybe the only short term improvement that is viable. Everything else is looking more long-term, and requires both a lot of effort and a lot of collaboration among ports. To the point that, as a practical matter, I'm not sure if the system would save more time than it would take to develop. That's not to say such a system won't be developed, but absent some sponsorship of the project or some highly motivated hackers, I don't know how we plan on getting there.

I just think this subject has come up numerous times, and each time the discussion ended without any improvements being made, so I worry that so long as we wait for the perfect system to come along, we're not going to build the good system that can make life better today. 

Thanks,

Kevin

> Regards,
> Maciej
> 
> 
> Regards,
> Maciej
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev