[webkit-dev] Towards a unified build system

[Adam Barth]

On Tue, Mar 1, 2011 at 2:46 PM, Mark Rowe <mrowe at apple.com> wrote:
> On 2011-03-01, at 14:29, Adam Barth wrote:
>> On Tue, Mar 1, 2011 at 2:18 PM, Mark Rowe <mrowe at apple.com> wrote:
>>> On 2011-03-01, at 14:06, Adam Barth wrote:
>>>> On Tue, Mar 1, 2011 at 1:57 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>>>>> On Mar 1, 2011, at 11:07 AM, Adam Barth wrote:
>>>>>> On Tue, Mar 1, 2011 at 9:33 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>>>>>>> Another consequence of step 3 is it would break submissions to Apple's central build system, since those pull from the repository with vanilla SVN and do not run special scripts afterwards.
>>>>>>
>>>>>> I don't fully understand the constraints of Apple's central build
>>>>>> system.  It might be worthwhile talking over the constraints with
>>>>>> someone who's an expert.
>>>>>>
>>>>>> If the requirements are that the entry point to the build system is an
>>>>>> xcodeproj, we can create an xcodeproj with an action that creates the
>>>>>> other xcodeproj files and then defers to the generated projects as
>>>>>> dependencies.  I haven't tested this approach, but I suspect we'll
>>>>>> find something that satisfies the Apple-internal constraints.
>>>>>
>>>>> It needs to be either an xcodeproj or a Makefile. Also, all tools required for building have to be either be part of the OS, or part of what is submitted for the given project. So if gyp scripts need to execute at build time, they'd either have to be checked in to the WebKit repository, or get added to Mac OS X.
>>>>
>>>> We can certainly use a Makefile to kick off the build.  The GYP system
>>>> itself is just a handful of BSD-licensed Python scripts, so I wouldn't
>>>> expect any trouble with checking them into svn.webkit.org.
>>>
>>> I don't think driving the build via a Makefile is a desirable option.  XBS (what Maciej has been referring to as Apple's central build system) knows how to override Xcode configuration settings for projects that build using Xcode. This ability is useful and sometimes necessary.  It's also not possible to use if a project builds using Make as Makefiles vary so much from project to project.
>>
>> This issue relates only to the "how and when to generate the projects"
>> phase.  It seems we have at least two feasible solutions (using
>> Makefiles and checking in generated projects), so it's mostly a matter
>> of thinking of more clever solutions and making desirability
>> trade-offs.
>
> Given that you've listed the Makefile as a feasible solution after what I've said above I can only assume that I misunderstand how you intend for it to be used.  Can you expand on how you see that working?

Oh, you wrote that it might not be desirable, not that it was
infeasible.  I'd classify the status quo as undesirable as well.  At
some point, we should have a higher bandwidth discussion about how to
make this work.

>>>>> I asked Mark Rowe about this and he says the current gyp-generated Xcode projects would not build in Apple's central build system at all, so I guess that hurdle would need to be cleared before we even replace the existing checked-in xcodeproj files. Mark is probably our top expert on how Apple's build system works.
>>>>
>>>> It would be helpful to understand why that's the case.  We can certain
>>>> improve the xcodeproj backend until it generates project files that
>>>> work with Apple's internal build system.  The current xcode backend is
>>>> only 1000 lines of Python.  We could re-write it from scratch if
>>>> needed.
>>>
>>> The initial issues that I observed when looking at JavaScriptCore was the Xcode project is generated in to a subdirectory.  This prevents "xcodebuild installsrc" from working correctly (only the files in directories below the Xcode project appear to be installed).
>>
>> Correct.  We're working in a subdirectory for the time begin to avoid
>> disrupting other folks.  Moving from a subdirectory to the main
>> directory is easy once we're ready.
>
> Until this particular issue is addressed it's going to be very difficult for anyone at Apple to evaluate the correctness of the generated projects.
>
>>>  I made some quick hacks to work around that and then ran in to the issue that the configurations in the generated JavaScriptCore.xcodeproj are not set up correctly (Production is no longer the default configuration, Production is based on DebugRelease.xcconfig rather than Base.xcconfig, etc.). There's also the fact that the generated Xcode project doesn't build the same content as the existing Xcode project (No jsc, testapi, minidom). At this point I gave up and stopped looking at it.
>>
>> Thanks for looking at the projects in detail.  The issues you're
>> running into are all things that we know we need to fix.  We're still
>> in the early experimental stages of step 1, which is why the generated
>> projects aren't complete yet.  At this point, I'm just happy they
>> build anything at all!
>
> One other thing that I just remembered is that the built framework doesn't include any headers!

Yes.  That's actually the bug I'm working on right now.  Mark Mentovai
recently added support to GYP for framework headers.  I need to teach
it how to understand private headers.  No one said this project would
be easy.  :)

> I suspected that the issues may be known, but it was suggested that I mention the issues I observed in case they were things that you weren't aware of.
>
> There are other issues that appear to be less specific to the particular JavaScriptCore project and more specific to gyp in general:
> 1) The generated Xcode project includes a number of configuration settings that are redundant given those that are present in the .xcconfig files (product name, Info.plist path, preprocessor macros, intermediates directory, to name a few).

It's slightly unclear to me what the trade-off is between storing
configuration information in xcconfig files and storing the same
information in the xcodeproj file itself.  Once we're further along, I
might ask for your help in identifying and fixing these sorts of
problems.

> 2) The generated Xcode project explicitly adds header include paths that point within the source tree.

This issue is connected with the header map configuration, as I
understand it.  We want to enable the header map, but that causes a
build error at the moment, which we need to resolve.

> 3) The generated Xcode project has script phases with strange names like 'Action "generate_derived_sources"' rather than a human-readable name like 'Generate Derived Sources'.

I've actually already posted a patch that should help with that issue.

> Those were a few things that I noticed from a quick cruise through the .pbxproj file.  I suspect that some of them may need to be addressed within gyp itself.

The gyp folks seem to be fairly responsive and willing to teach me how
to improve GYP, so hopefully we can get these issues squared away.

Thanks,
Adam