[webkit-dev] PSA: Changes in the recommended GTK/WPE developer workflow

[Philippe Normand]

Hi,

Until now the most recommended way to work on the GTK and WPE ports was
to use a custom JHBuild[0] moduleset ([1][2]) for build/runtime
dependencies. This setup was deployed almost 10 years ago [3] and while
it is an opt-in approach, it is highly recommended, especially if you
want to run layout tests locally.

If you ever executed Tools/Scripts/update-webkitgtk-libs or
Tools/Scripts/update-webkitwpe-libs it means you currently have a
DependenciesGTK and/or DependenciesWPE folder in WebKitBuild/ and that
your workflow is JHBuild-based.

JHBuild has various shortcomings, even though it served us well all
these years:

- if the moduleset is modified, the bots remove the whole
Dependencies{GTK,WPE} folders and rebuild everything from scratch
- no cross-compilation support
- no clear separation between the host system and the JHBuild-based
sysroot
- poor system requirements management (See the install-dependencies
scripts in Tools/{wpe,gtk})
- time lost rebuilding the dependencies (can take up to an hour on my
build machine)

So in order to improve our lives a bit, we decided to try a new
workflow, this time based on Flatpak[4]. Instead of having every
developer locally build the dependencies, the built sysroot will be
packaged in a Flatpak SDK/Runtime and downloaded to the developer
machine. Currently the SDK is built for X86_64 but additional
architectures can be supported (ARMv7, Aarch64 at least).

The advantages of this new approach:

- cross-compilation can easily be achieved
- clear separation between the host system and the SDK sysroot
- the SDK runs in a sandbox
- unified toolchain for WebKit build (currently both GCC 9.2.0 and
clang 8.0.1)
- no time lost rebuilding dependencies :)
- consistent layout tests coverage across different hosts
- separate build directories for each port (example:
WebKitBuild/GTK/Release
  mounted as /app/WebKit/WebKitBuild/Release in the SDK sandbox)

As a bonus, this setup allows for:

- integration with sccache, for faster WebKit builds
- builtin IceCC support without additional manual steps (you still need
a scheduler running on the host system though)

The only disadvantage of this new approach is that hacking on
dependencies is currently not trivial to accomplish. We plan to enable
this most likely via the Flapjack[5] tool.

Once the patch from bug#205658 [6] lands, this workflow will be
available. How do I use it? Easy:

1. Make sure your flatpak version is recent enough, 1.4.4 is the
minimum version we support. Backports for Debian stable are available
and for Ubuntu LTS, a PPA is available. See the Flathub instructions
(though you don't need to manually add the flathub remote) [7].

2. Run update-webkitgtk-libs or update-webkitwpe-libs as usual. This
will download and install the SDK in WebKitBuild/UserFlatpak/

3. Run build-webkit as usual

The SDK will be used when running the tests (API, layout, webkitpy,
etc) and the MiniBrowser.

For the time being we keep the JHBuild workflow in the SVN, although if
everything goes well, it will be removed in the coming months. Hence we
encourage all GTK/WPE developers to try this new workflow!

If you still want to keep using JHBuild, make sure to set the
WEBKIT_JHBUILD=1 environment variable in your shell. But keep in mind
that we intend to remove support for JHBuild if Flatpak works as
expected (so please try to test it, and report any issue with it). The
GTK/WPE buildbots and EWS are currently running with JHBuild and will
be switched one-by-one to Flatpak soon.

The SDK sources are currently hosted on Github[8], but we might move
them to WebKit's SVN.

Philippe

[0] https://developer.gnome.org/jhbuild/stable/
[1] https://trac.webkit.org/browser/trunk/Tools/gtk/jhbuild.modules
[2] https://trac.webkit.org/browser/trunk/Tools/wpe/jhbuild.modules
[3] https://bugs.webkit.org/show_bug.cgi?id=73425
[4] https://flatpak.org/
[5] https://github.com/endlessm/flapjack
[6] https://bugs.webkit.org/show_bug.cgi?id=205658
[7] https://flatpak.org/setup/
[8] https://github.com/igalia/webkit-flatpak-sdk