[webkit-dev] Towards a unified build system

[Mark Rowe]

On 2011-03-01, at 14:29, Adam Barth wrote:

> On Tue, Mar 1, 2011 at 2:18 PM, Mark Rowe <mrowe at apple.com> wrote:
>> On 2011-03-01, at 14:06, Adam Barth wrote:
>>> On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>>> On Mar 1, 2011, at 11:07 AM, Adam Barth wrote:
>>>>> On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>>>>>> Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards.
>>>>> 
>>>>> I don't fully understand the constraints of Apple's central build
>>>>> system.  It might be worthwhile talking over the constraints with
>>>>> someone who's an expert.
>>>>> 
>>>>> If the requirements are that the entry point to the build system is an
>>>>> xcodeproj, we can create an xcodeproj with an action that creates the
>>>>> other xcodeproj files and then defers to the generated projects as
>>>>> dependencies.  I haven't tested this approach, but I suspect we'll
>>>>> find something that satisfies the Apple-internal constraints.
>>>> 
>>>> It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X.
>>> 
>>> We can certainly use a Makefile to kick off the build.  The GYP system
>>> itself is just a handful of BSD-licensed Python scripts, so I wouldn't
>>> expect any trouble with checking them into svn.webkit.org.
>> 
>> I don't think driving the build via a Makefile is a desirable option.  XBS (what Maciej has been referring to as Apple's central build system) knows how to override Xcode configuration settings for projects that build using Xcode. This ability is useful and sometimes necessary.  It's also not possible to use if a project builds using Make as Makefiles vary so much from project to project.
> 
> This issue relates only to the "how and when to generate the projects"
> phase.  It seems we have at least two feasible solutions (using
> Makefiles and checking in generated projects), so it's mostly a matter
> of thinking of more clever solutions and making desirability
> trade-offs.

Given that you've listed the Makefile as a feasible solution after what I've said above I can only assume that I misunderstand how you intend for it to be used.  Can you expand on how you see that working?

>>>> I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works.
>>> 
>>> It would be helpful to understand why that's the case.  We can certain
>>> improve the xcodeproj backend until it generates project files that
>>> work with Apple's internal build system.  The current xcode backend is
>>> only 1000 lines of Python.  We could re-write it from scratch if
>>> needed.
>> 
>> The initial issues that I observed when looking at JavaScriptCore was the Xcode project is generated in to a subdirectory.  This prevents "xcodebuild installsrc" from working correctly (only the files in directories below the Xcode project appear to be installed).
> 
> Correct.  We're working in a subdirectory for the time begin to avoid
> disrupting other folks.  Moving from a subdirectory to the main
> directory is easy once we're ready.

Until this particular issue is addressed it's going to be very difficult for anyone at Apple to evaluate the correctness of the generated projects.

>>  I made some quick hacks to work around that and then ran in to the issue that the configurations in the generated JavaScriptCore.xcodeproj are not set up correctly (Production is no longer the default configuration, Production is based on DebugRelease.xcconfig rather than Base.xcconfig, etc.). There's also the fact that the generated Xcode project doesn't build the same content as the existing Xcode project (No jsc, testapi, minidom). At this point I gave up and stopped looking at it.
> 
> Thanks for looking at the projects in detail.  The issues you're
> running into are all things that we know we need to fix.  We're still
> in the early experimental stages of step 1, which is why the generated
> projects aren't complete yet.  At this point, I'm just happy they
> build anything at all!

One other thing that I just remembered is that the built framework doesn't include any headers!

I suspected that the issues may be known, but it was suggested that I mention the issues I observed in case they were things that you weren't aware of. 

There are other issues that appear to be less specific to the particular JavaScriptCore project and more specific to gyp in general:
1) The generated Xcode project includes a number of configuration settings that are redundant given those that are present in the .xcconfig files (product name, Info.plist path, preprocessor macros, intermediates directory, to name a few).
2) The generated Xcode project explicitly adds header include paths that point within the source tree.
3) The generated Xcode project has script phases with strange names like 'Action "generate_derived_sources"' rather than a human-readable name like 'Generate Derived Sources'.

Those were a few things that I noticed from a quick cruise through the .pbxproj file.  I suspect that some of them may need to be addressed within gyp itself.

- Mark