[webkit-dev] Better syntax for GYP (Was Common build system; was Re: WebKit Wishes)
gregory.szorc at gmail.com
Tue Feb 5 14:59:54 PST 2013
I work on the build system for Firefox and have some experience that
might be worth sharing.
We (Mozilla) evaluated GYP as the replacement for Makefile.in files.
While we generally love the concept of GYP (assemble data structures
then transform them N ways to whatever build backend/tool format you
want), we didn't really care for GYP's frontend file format. A "static"
file format like JSON, YAML, XML, etc just doesn't feel right,
especially when you need to do things like "if X & Y then do these things."
After a significant conversation on one of our main developer lists 
(many parts relevant to this post), we flirted with ini-like manifests
and Lua (it was invented as a config file language after all) but
eventually chose sandboxed Python. Once we decided to go with a real
programming language, the choice of Python was easy, as that is what
most of our tools are written in.
Instead of Makefile.in or .gyp files, we have what are called
"moz.build" files. These are actually Python scripts executed in a
highly limited Python environment (no importing, no I/O, etc). We define
special functions and variables. At the end of each file's execution, we
extract values from the symbol table then construct data structures
representing the build configuration. At this point, we essentially have
arrived at the backend portion of GYP.
See  for slides with some examples. A technical readme exists at .
The main code lives at  and . The sandboxing code  is the real
bread and butter. With relative ease it can be extracted from the
Firefox source tree (we'd like to eventually have it live on PyPI).
Unfortunately, we haven't yet landed the initial switchover, so I can't
tell you how the experience has been received. Having personally
converted large portions of our source tree to use moz.build files, I
can say that it feels much better writing "real" code as opposed to
shoehorning procedural logic into something like YAML.
We have aspirations of eventually hooking our moz.build-derived data
structures into GYP (why reinvent the build backend generation wheel).
If that works out well, we speculated we could contribute the Python
sandboxing code to GYP as an alternate frontend file format or at least
make it available as a standalone package. But this is so far out that
it's only wishful thinking. If WebKit or some other large GYP-using
project were interested, I'd love to explore possibilities of unifying
I don't actively follow this mailing list, so please CC me on any
replies. You can also find me as gps on irc.mozilla.org (try #build).
gregory.szorc at gmail.com
On 2/5/13 2:01 PM, Ryosuke Niwa wrote:
> On Tue, Feb 5, 2013 at 1:36 PM, Mark Mentovai <mark at chromium.org
> <mailto:mark at chromium.org>> wrote:
> In the end, I don’t think that the actual syntax (as opposed to the
> structured data that it contains) is all that important. GYP input
> files are fairly simple structured data consisting of very few
> types: dicts, lists, strings, and infrequent bools and ints. Any
> extant or nascent format that can handle these is a candidate. If
> there’s some kind of convergence on a beautiful one that’s easy to
> maintain and quick to parse, there’s no reason GYP can’t be taught
> to read it, even alongside the existing JSON-esque format that’s fed
> to the Python interpreter. If this is the only thing standing
> between a project and its adopting GYP, then it’s a really easy one
> to get over.
> The syntax has always been the biggest problem when it comes to adopting
> GYP in WebKit.
> Let's take the following as an example as picked by Maciej
> 'action_name': 'XMLNames',
> 'inputs': [
> '../dom/make_names.pl <http://make_names.pl>',
> '../xml/xmlattrs.in <http://xmlattrs.in>',
> 'outputs': [
> 'action': [
> '--extraDefines', '<(feature_defines)'
> 'msvs_cygwin_shell': 1,
> Using a YAML-like syntax, we can rewrite it as:
> <(SHARED_INTERMEDIATE_DIR)/../xml/xmlattrs.in <http://xmlattrs.in>
> python scripts/action_makenames.py <@(_outputs) -- <@(_inputs)
> -- --extraDefines <(feature_defines)
> msvs_cygwin_shell: True
> To me, this YAML-like syntax reads significantly better.
> Now, I don't like all the magic strings like
> "<(SHARED_INTERMEDIATE_DIR)/../" and "<@(_outputs) -- <@(_inputs) --
> --extraDefines <(feature_defines)". Can someone with build system
> knowlede prettify them?
> - R. Niwa
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
More information about the webkit-dev