Build system complexity
Are there currently any plans for simplifying the situation regarding build systems? I haven't seen any threads for a while, which I assume means no. Is there any low hanging fruit out there? Since many of the build systems are little more than lists of files, it really seems like we should be able to do some sort of consolidation. Or reduce the process down to updating one file and then running a script that updates/generates the rest. Currently, I cringe every time I find out I need to add a new file. In addition, has anyone ever looked at simplifying the mac port's xcode project? It's _by far_ the heaviest burden on the project given that you pretty much need to use xcode (which is mac only...no other port requires this), exported linker symbols are in a separate file, extra effort to expose a new file in WTF to WebCore, extra effort to expose a new file in WebCore to WebKit, etc. Has anyone recently looked at how the mac build could be simplified--especially from the perspective of contributors whose main development platform isn't a mac? Thanks! Jeremy
In addition, has anyone ever looked at simplifying the mac port's xcode project? It's _by far_ the heaviest burden on the project given that you pretty much need to use xcode (which is mac only...no other port requires this), exported linker symbols are in a separate file, extra effort to expose a new file in WTF to WebCore, extra effort to expose a new file in WebCore to WebKit, etc. Has anyone recently looked at how the mac build could be simplified--especially from the perspective of contributors whose main development platform isn't a mac?
i completely agree with jeremy. is it possible to at least drop the cryptic hashcodes/timestamps? without them, the .xcodeproj files should at least be editable by hand. dumi
On Aug 12, 2010, at 3:54 AM, Dumitru Daniliuc <dumi@chromium.org> wrote:
i completely agree with jeremy. is it possible to at least drop the cryptic hashcodes/timestamps? without them, the .xcodeproj files should at least be editable by hand.
Doesn't gyp already generate Xcode projects for Chrome? I think the issue is that gyp can't generate replacement project files for Apple's Mac port or other build systems yet. That was my take-away from the last discussion--that gyp needed to be enhanced so that all build systems could be generated, making addition or removal of source files a trivial task. As far as Jeremy's other points, perhaps bugs should be filed about each item so that more specific discussion can take place on each topic. Dave
On Thu, Aug 12, 2010 at 7:18 AM, David Kilzer <ddkilzer@webkit.org> wrote:
On Aug 12, 2010, at 3:54 AM, Dumitru Daniliuc <dumi@chromium.org> wrote:
i completely agree with jeremy. is it possible to at least drop the cryptic hashcodes/timestamps? without them, the .xcodeproj files should at least be editable by hand.
Doesn't gyp already generate Xcode projects for Chrome? I think the issue is that gyp can't generate replacement project files for Apple's Mac port or other build systems yet. That was my take-away from the last discussion--that gyp needed to be enhanced so that all build systems could be generated, making addition or removal of source files a trivial task.
Chrome has been using GYP for some time and it's pretty stable. I suspect that anyone trying to port the Mac port's xcode project to it will run into some bugs and/or need to build some additional features into it, but it is quite stable. And the GYP guys are very friendly people though, so I bet they'd be happy to help with any such problems. It's also worth noting that GYP was designed from the start to allow a project to move over to it slowly (you can have custom projects depend on GYP projects and vice versa). But moving to GYP is definitely not the only way to solve this issue--and quite possibly not the best way. I suspect that there are also many smaller steps that could be taken that'd have a big impact. For example, coming up with ways to generate sane/informative error messages for when someone doesn't export some symbol/header file properly would awesome and doesn't require changing the entire build system. Or creating some script that can add files to the xcode project. J
Let me re-iterate (because even some co-workers are getting confused): I'm not secretly trying to get the mac port to start using GYP. (Or saying it's a bad idea either, mind you.) I'm just concerned that the process associated with adding a file (or adding a dependency between projects on the mac port) is out of hand--and has been for some time. And I'm hoping we can come to some sort of conclusion that'll start the project moving in the right direction--even if it's slowly. That's it. I promise! J On Thu, Aug 12, 2010 at 3:37 PM, Jeremy Orlow <jorlow@chromium.org> wrote:
On Thu, Aug 12, 2010 at 7:18 AM, David Kilzer <ddkilzer@webkit.org> wrote:
On Aug 12, 2010, at 3:54 AM, Dumitru Daniliuc <dumi@chromium.org> wrote:
i completely agree with jeremy. is it possible to at least drop the cryptic hashcodes/timestamps? without them, the .xcodeproj files should at least be editable by hand.
Doesn't gyp already generate Xcode projects for Chrome? I think the issue is that gyp can't generate replacement project files for Apple's Mac port or other build systems yet. That was my take-away from the last discussion--that gyp needed to be enhanced so that all build systems could be generated, making addition or removal of source files a trivial task.
Chrome has been using GYP for some time and it's pretty stable. I suspect that anyone trying to port the Mac port's xcode project to it will run into some bugs and/or need to build some additional features into it, but it is quite stable. And the GYP guys are very friendly people though, so I bet they'd be happy to help with any such problems. It's also worth noting that GYP was designed from the start to allow a project to move over to it slowly (you can have custom projects depend on GYP projects and vice versa).
But moving to GYP is definitely not the only way to solve this issue--and quite possibly not the best way. I suspect that there are also many smaller steps that could be taken that'd have a big impact. For example, coming up with ways to generate sane/informative error messages for when someone doesn't export some symbol/header file properly would awesome and doesn't require changing the entire build system. Or creating some script that can add files to the xcode project.
J
Hi Jeremy, On Aug 12, 2010, at 7:37 AM, Jeremy Orlow wrote:
On Thu, Aug 12, 2010 at 7:18 AM, David Kilzer <ddkilzer@webkit.org> wrote: On Aug 12, 2010, at 3:54 AM, Dumitru Daniliuc <dumi@chromium.org> wrote:
i completely agree with jeremy. is it possible to at least drop the cryptic hashcodes/timestamps? without them, the .xcodeproj files should at least be editable by hand.
Doesn't gyp already generate Xcode projects for Chrome? I think the issue is that gyp can't generate replacement project files for Apple's Mac port or other build systems yet. That was my take-away from the last discussion--that gyp needed to be enhanced so that all build systems could be generated, making addition or removal of source files a trivial task.
Chrome has been using GYP for some time and it's pretty stable. I suspect that anyone trying to port the Mac port's xcode project to it will run into some bugs and/or need to build some additional features into it, but it is quite stable. And the GYP guys are very friendly people though, so I bet they'd be happy to help with any such problems. It's also worth noting that GYP was designed from the start to allow a project to move over to it slowly (you can have custom projects depend on GYP projects and vice versa).
But moving to GYP is definitely not the only way to solve this issue--and quite possibly not the best way. I suspect that there are also many smaller steps that could be taken that'd have a big impact. For example, coming up with ways to generate sane/informative error messages for when someone doesn't export some symbol/header file properly would awesome and doesn't require changing the entire build system. Or creating some script that can add files to the xcode project.
One project I've been meaning to hack on once I take care of some higher priority work I'm doing is to update WebKitTools/Scripts/update-sources-list.py to pull sources from GYP instead of the code in there to pull sources from (now defunct) Bakefile build system. The idea here is that we can make changes to the source list in one place (say, the GYP files), then run this script and it will update the file lists for other ports that use text-based lists, like GTK, Qt and any ports using CMake. This I think would make updating a lot more straightforward and reduce the need for manually making the same change in 6 different build systems. Adding XCode and MSVC support certainly would be possible too, but if ports using those files are still interested in doing a switch to GYP in the near future, it may not make much sense to bother with it. I've also been working on https://bugs.webkit.org/show_bug.cgi?id=27551, which would allow us to stop using export symbol definition files and instead have the export info in the headers instead, so that all ports can use them rather than having each port define its own export symbols file. That's another thing that I think could reduce a lot of redundant maintenance. Regards, Kevin
J _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Eric and I discussed this recently. We think the easiest path forward is to try to remove build systems incrementally. Each build system we remove makes it easier to hack on the project because you have one less build system to worry about. The easiest build system to remove is probably the Visual Studio build system. There's some complexity there because several ports use the vcproj files (e.g., apple-win, win-cairo, and wince as I recall). My understanding (based on past emails) is that either CMake or GYP would be workable. I'm not sure there's anything blocking this work. We just need someone to be interested enough in making it happen to do it. Adam On Thu, Aug 12, 2010 at 2:53 AM, Jeremy Orlow <jorlow@chromium.org> wrote:
Are there currently any plans for simplifying the situation regarding build systems? I haven't seen any threads for a while, which I assume means no. Is there any low hanging fruit out there? Since many of the build systems are little more than lists of files, it really seems like we should be able to do some sort of consolidation. Or reduce the process down to updating one file and then running a script that updates/generates the rest. Currently, I cringe every time I find out I need to add a new file. In addition, has anyone ever looked at simplifying the mac port's xcode project? It's _by far_ the heaviest burden on the project given that you pretty much need to use xcode (which is mac only...no other port requires this), exported linker symbols are in a separate file, extra effort to expose a new file in WTF to WebCore, extra effort to expose a new file in WebCore to WebKit, etc. Has anyone recently looked at how the mac build could be simplified--especially from the perspective of contributors whose main development platform isn't a mac? Thanks! Jeremy _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
An added benefit of removing the VS build system first would be that we'd be able to build using VS2005 or VS2008. On Thu, Aug 12, 2010 at 10:36 AM, Adam Barth <abarth@webkit.org> wrote:
Eric and I discussed this recently. We think the easiest path forward is to try to remove build systems incrementally. Each build system we remove makes it easier to hack on the project because you have one less build system to worry about. The easiest build system to remove is probably the Visual Studio build system. There's some complexity there because several ports use the vcproj files (e.g., apple-win, win-cairo, and wince as I recall). My understanding (based on past emails) is that either CMake or GYP would be workable.
I'm not sure there's anything blocking this work. We just need someone to be interested enough in making it happen to do it.
Adam
On Thu, Aug 12, 2010 at 2:53 AM, Jeremy Orlow <jorlow@chromium.org> wrote:
Are there currently any plans for simplifying the situation regarding build systems? I haven't seen any threads for a while, which I assume means no. Is there any low hanging fruit out there? Since many of the build systems are little more than lists of files, it really seems like we should be able to do some sort of consolidation. Or reduce the process down to updating one file and then running a script that updates/generates the rest. Currently, I cringe every time I find out I need to add a new file. In addition, has anyone ever looked at simplifying the mac port's xcode project? It's _by far_ the heaviest burden on the project given that you pretty much need to use xcode (which is mac only...no other port requires this), exported linker symbols are in a separate file, extra effort to expose a new file in WTF to WebCore, extra effort to expose a new file in WebCore to WebKit, etc. Has anyone recently looked at how the mac build could be simplified--especially from the perspective of contributors whose main development platform isn't a mac? Thanks! Jeremy _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Eric and I discussed this recently. We think the easiest path forward is to try to remove build systems incrementally. Each build system we remove makes it easier to hack on the project because you have one less build system to worry about. The easiest build system to remove is probably the Visual Studio build system. There's some complexity there because several ports use the vcproj files (e.g., apple-win, win-cairo, and wince as I recall). My understanding (based on past emails) is that either CMake or GYP would be workable. Last CMake discussion message ended with [1]. I strongly prefer CMake, because GYP is a Chromium tool ("... a tool that is highly specific to Chromium's specific use case ..." [2])
I'm not sure there's anything blocking this work. We just need someone to be interested enough in making it happen to do it. I work already on a general CMake buildsystem [3]. Until I get a r+, the patch doesn't aplly any more, what makes it very hard to make progress (i took 3 months for the last patch). I use CMake for the WinCE port too [4]. It should be very easy to add additional ports to the CMake system. You only need to add CMakeLists<PORT>.txt files, which contain the plarform specific files (see [5] for additional sample).
IMHO it's no big task to port wx and gtk to CMake. For the Windows port(s) it's a little bit harder if you want to detect the 3rdparty stuff in a clean way (because there is no general "/usr/include"). There is already a bug to use CMake for BREWMP: [6] If someone is interested in switching to CMake I'd like to help. [1] https://lists.webkit.org/pipermail/webkit-dev/2010-April/012662.html [2] http://code.google.com/p/gyp/wiki/GypLanguageSpecification#Background [3] https://bugs.webkit.org/show_bug.cgi?id=43906 [4] http://trac.webkit.org/wiki/WinCE [5] http://gitorious.org/+wincewebkit- developers/webkit/wincewebkit/trees/wincegdi [6] https://bugs.webkit.org/show_bug.cgi?id=39398 - Patrick
On Aug 12, 2010, at 2:53 AM, Jeremy Orlow wrote:
Are there currently any plans for simplifying the situation regarding build systems? I haven't seen any threads for a while, which I assume means no.
Is there any low hanging fruit out there? Since many of the build systems are little more than lists of files, it really seems like we should be able to do some sort of consolidation. Or reduce the process down to updating one file and then running a script that updates/generates the rest. Currently, I cringe every time I find out I need to add a new file.
In addition, has anyone ever looked at simplifying the mac port's xcode project? It's _by far_ the heaviest burden on the project given that you pretty much need to use xcode (which is mac only...no other port requires this), exported linker symbols are in a separate file, extra effort to expose a new file in WTF to WebCore, extra effort to expose a new file in WebCore to WebKit, etc. Has anyone recently looked at how the mac build could be simplified--especially from the perspective of contributors whose main development platform isn't a mac?
I think we should switch over to export control specified in the headers and not use .exp files any more. WebKit2.framework builds on all its target platforms with no external export control file. The hardest case for this is probably WebCore, though, and I'm not sure how practical it is to do incrementally. Regards, Maciej
Hi Maciej, On Aug 13, 2010, at 12:34 AM, Maciej Stachowiak wrote:
On Aug 12, 2010, at 2:53 AM, Jeremy Orlow wrote:
Are there currently any plans for simplifying the situation regarding build systems? I haven't seen any threads for a while, which I assume means no.
Is there any low hanging fruit out there? Since many of the build systems are little more than lists of files, it really seems like we should be able to do some sort of consolidation. Or reduce the process down to updating one file and then running a script that updates/generates the rest. Currently, I cringe every time I find out I need to add a new file.
In addition, has anyone ever looked at simplifying the mac port's xcode project? It's _by far_ the heaviest burden on the project given that you pretty much need to use xcode (which is mac only...no other port requires this), exported linker symbols are in a separate file, extra effort to expose a new file in WTF to WebCore, extra effort to expose a new file in WebCore to WebKit, etc. Has anyone recently looked at how the mac build could be simplified--especially from the perspective of contributors whose main development platform isn't a mac?
I think we should switch over to export control specified in the headers and not use .exp files any more. WebKit2.framework builds on all its target platforms with no external export control file. The hardest case for this is probably WebCore, though, and I'm not sure how practical it is to do incrementally.
Actually, this reminds me of one thing I've been meaning to ask about this regarding moving JSCore export definitions into the headers. The export symbol definitions (such as JS_EXPORTDATA, JS_EXPORT_PRIVATE, etc.) are defined multiple ways in multiple places (e.g. various project's config.h headers), and I was hoping we could move towards having some sort of WTF_EXPORT and WTF_IMPORT macros, then set macros like JS_EXPORTDATA based on them, so that we're not continually copying and pasting the same macro setting logic into each project's config.h or somewhere else. Kind of like a project-wide equivalent to JSBase.h for JSCore public API. You can see this quite well how many places these macros are copied over to by looking at the last patch for https://bugs.webkit.org/show_bug.cgi?id=27551, but that's a pretty large patch, so a simpler case which clearly shows how we need to copy and paste these macros each time a new component needs export symbol control can be found in a patch I recently submitted here: https://bugs.webkit.org/show_bug.cgi?id=43204 One thing that seems to make consolidating these macros into one location tricky is that WTF seems the most suitable place for it, but it is still technically a private implementation detail of JSCore, and there seems to be with some ports a great deal of effort put into isolating each project's headers and sources from the other projects. (With the Windows build particularly, I noticed a lot of copying of headers and even some implementation files from one project to another using the build directory as an intermediary.) I'd like to have some file that could be included by everything from JSCore to DumpRenderTree and, at least indirectly, by client apps (e.g. when I use those macros in wxWebKit API), without having that sort of copying set up in all the ports. Is there currently an appropriate place for such a thing to go? If not, any suggestions on the best way to approach this? Or am I missing some obvious and simple solution here? Thanks, Kevin
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
participants (8)
-
Adam Barth
-
David Kilzer
-
Dumitru Daniliuc
-
Jeremy Orlow
-
Kevin Ollivier
-
Maciej Stachowiak
-
Ojan Vafai
-
Patrick Roland Gansterer