[webkit-dev] Better syntax for GYP (Was Common build system; was Re: WebKit Wishes)

Tue Feb 5 14:59:54 PST 2013

I work on the build system for Firefox and have some experience that 
might be worth sharing.

We (Mozilla) evaluated GYP as the replacement for Makefile.in files. 
While we generally love the concept of GYP (assemble data structures 
then transform them N ways to whatever build backend/tool format you 
want), we didn't really care for GYP's frontend file format. A "static" 
file format like JSON, YAML, XML, etc just doesn't feel right, 
especially when you need to do things like "if X & Y then do these things."

After a significant conversation on one of our main developer lists [1] 
(many parts relevant to this post), we flirted with ini-like manifests 
and Lua (it was invented as a config file language after all) but 
eventually chose sandboxed Python. Once we decided to go with a real 
programming language, the choice of Python was easy, as that is what 
most of our tools are written in.

Instead of Makefile.in or .gyp files, we have what are called 
"moz.build" files. These are actually Python scripts executed in a 
highly limited Python environment (no importing, no I/O, etc). We define 
special functions and variables. At the end of each file's execution, we 
extract values from the symbol table then construct data structures 
representing the build configuration. At this point, we essentially have 
arrived at the backend portion of GYP.

See [2] for slides with some examples. A technical readme exists at [3]. 
The main code lives at [4] and [5]. The sandboxing code [4] is the real 
bread and butter. With relative ease it can be extracted from the 
Firefox source tree (we'd like to eventually have it live on PyPI).

Unfortunately, we haven't yet landed the initial switchover, so I can't 
tell you how the experience has been received. Having personally 
converted large portions of our source tree to use moz.build files, I 
can say that it feels much better writing "real" code as opposed to 
shoehorning procedural logic into something like YAML.

We have aspirations of eventually hooking our moz.build-derived data 
structures into GYP (why reinvent the build backend generation wheel). 
If that works out well, we speculated we could contribute the Python 
sandboxing code to GYP as an alternate frontend file format or at least 
make it available as a standalone package. But this is so far out that 
it's only wishful thinking. If WebKit or some other large GYP-using 
project were interested, I'd love to explore possibilities of unifying 
solutions.

I don't actively follow this mailing list, so please CC me on any 
replies. You can also find me as gps on irc.mozilla.org (try #build).

Gregory Szorc
gregory.szorc at gmail.com

[1] 
https://groups.google.com/d/topic/mozilla.dev.platform/SACOnl-avMs/discussion
[2] 
http://gregoryszorc.com/presentations/2012-11-29-firefox-build-system/#41
[3] 
https://hg.mozilla.org/mozilla-central/file/1eab3e0d7f53/python/mozbuild/mozbuild/frontend/README.rst
[4] 
https://hg.mozilla.org/mozilla-central/file/1eab3e0d7f53/python/mozbuild/mozbuild/frontend/sandbox.py
[5] 
https://hg.mozilla.org/mozilla-central/file/1eab3e0d7f53/python/mozbuild/mozbuild/frontend/reader.py

On 2/5/13 2:01 PM, Ryosuke Niwa wrote:
> On Tue, Feb 5, 2013 at 1:36 PM, Mark Mentovai <mark at chromium.org
> <mailto:mark at chromium.org>> wrote:
>
>     In the end, I don’t think that the actual syntax (as opposed to the
>     structured data that it contains) is all that important. GYP input
>     files are fairly simple structured data consisting of very few
>     types: dicts, lists, strings, and infrequent bools and ints. Any
>     extant or nascent format that can handle these is a candidate. If
>     there’s some kind of convergence on a beautiful one that’s easy to
>     maintain and quick to parse, there’s no reason GYP can’t be taught
>     to read it, even alongside the existing JSON-esque format that’s fed
>     to the Python interpreter. If this is the only thing standing
>     between a project and its adopting GYP, then it’s a really easy one
>     to get over.
>
>
> The syntax has always been the biggest problem when it comes to adopting
> GYP in WebKit.
>
> Let's take the following as an example as picked by Maciej
> {
>    'action_name': 'XMLNames',
>    'inputs': [
>      '../dom/make_names.pl <http://make_names.pl>',
>      '../xml/xmlattrs.in <http://xmlattrs.in>',
>    ],
>    'outputs': [
>      '<(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.cpp',
>      '<(SHARED_INTERMEDIATE_DIR)/webkit/XMLNames.h',
>    ],
>    'action': [
>      'python',
>      'scripts/action_makenames.py',
>      '<@(_outputs)',
>      '--',
>      '<@(_inputs)',
>      '--',
>      '--extraDefines', '<(feature_defines)'
>    ],
>    'msvs_cygwin_shell': 1,
> },
>
> Using a YAML-like syntax, we can rewrite it as:
> XMLName:
>      inputs:
>          <(SHARED_INTERMEDIATE_DIR)/../dom/make_names.pl
> <http://make_names.pl>
>          <(SHARED_INTERMEDIATE_DIR)/../xml/xmlattrs.in <http://xmlattrs.in>
>      outputs:
>          XMLNames.cpp
>          XMLNames.h
>      action:
>          python scripts/action_makenames.py <@(_outputs) -- <@(_inputs)
> -- --extraDefines <(feature_defines)
>      msvs_cygwin_shell: True
>
> To me, this YAML-like syntax reads significantly better.
>
> Now, I don't like all the magic strings like
> "<(SHARED_INTERMEDIATE_DIR)/../" and  "<@(_outputs) -- <@(_inputs) --
> --extraDefines <(feature_defines)". Can someone with build system
> knowlede prettify them?
>
> - R. Niwa
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>