[webkit-dev] PSA: Changes in the recommended GTK/WPE developer workflow

[Philippe Normand]

Hi,

A few updates about this topic. Read below!

On Tue, 2020-03-17 at 17:31 +0000, Philippe Normand wrote:
> Hi,
> 
> Until now the most recommended way to work on the GTK and WPE ports
> was
> to use a custom JHBuild[0] moduleset ([1][2]) for build/runtime
> dependencies. This setup was deployed almost 10 years ago [3] and
> while
> it is an opt-in approach, it is highly recommended, especially if you
> want to run layout tests locally.
> 
> If you ever executed Tools/Scripts/update-webkitgtk-libs or
> Tools/Scripts/update-webkitwpe-libs it means you currently have a
> DependenciesGTK and/or DependenciesWPE folder in WebKitBuild/ and
> that
> your workflow is JHBuild-based.
> 
> JHBuild has various shortcomings, even though it served us well all
> these years:
> 
> - if the moduleset is modified, the bots remove the whole
> Dependencies{GTK,WPE} folders and rebuild everything from scratch
> - no cross-compilation support
> - no clear separation between the host system and the JHBuild-based
> sysroot
> - poor system requirements management (See the install-dependencies
> scripts in Tools/{wpe,gtk})
> - time lost rebuilding the dependencies (can take up to an hour on my
> build machine)
> 
> So in order to improve our lives a bit, we decided to try a new
> workflow, this time based on Flatpak[4]. Instead of having every
> developer locally build the dependencies, the built sysroot will be
> packaged in a Flatpak SDK/Runtime and downloaded to the developer
> machine. Currently the SDK is built for X86_64 but additional
> architectures can be supported (ARMv7, Aarch64 at least).
> 
> The advantages of this new approach:
> 
> - cross-compilation can easily be achieved
> - clear separation between the host system and the SDK sysroot
> - the SDK runs in a sandbox
> - unified toolchain for WebKit build (currently both GCC 9.2.0 and
> clang 8.0.1)
> - no time lost rebuilding dependencies :)
> - consistent layout tests coverage across different hosts
> - separate build directories for each port (example:
> WebKitBuild/GTK/Release
>   mounted as /app/WebKit/WebKitBuild/Release in the SDK sandbox)
> 
> As a bonus, this setup allows for:
> 
> - integration with sccache, for faster WebKit builds
> - builtin IceCC support without additional manual steps (you still
> need
> a scheduler running on the host system though)
> 
> The only disadvantage of this new approach is that hacking on
> dependencies is currently not trivial to accomplish. We plan to
> enable
> this most likely via the Flapjack[5] tool.
> 

Flapjack was discarded, for now, because Buildstream provides a good
workflow already.

> Once the patch from bug#205658 [6] lands, this workflow will be
> available. How do I use it? Easy:
> 
> 1. Make sure your flatpak version is recent enough, 1.4.4 is the
> minimum version we support. Backports for Debian stable are available
> and for Ubuntu LTS, a PPA is available. See the Flathub instructions
> (though you don't need to manually add the flathub remote) [7].
> 
> 2. Run update-webkitgtk-libs or update-webkitwpe-libs as usual. This
> will download and install the SDK in WebKitBuild/UserFlatpak/
> 
> 3. Run build-webkit as usual
> 
> The SDK will be used when running the tests (API, layout, webkitpy,
> etc) and the MiniBrowser.
> 
> For the time being we keep the JHBuild workflow in the SVN, although
> if
> everything goes well, it will be removed in the coming months. Hence
> we
> encourage all GTK/WPE developers to try this new workflow!
> 
> If you still want to keep using JHBuild, make sure to set the
> WEBKIT_JHBUILD=1 environment variable in your shell. But keep in mind
> that we intend to remove support for JHBuild if Flatpak works as
> expected (so please try to test it, and report any issue with it).
> The
> GTK/WPE buildbots and EWS are currently running with JHBuild and will
> be switched one-by-one to Flatpak soon.
> 

The bots now use the SDK, thanks to Diego Pino, Lauro Mauro and Carlos
López. Many thanks to them :)

> The SDK sources are currently hosted on Github[8], but we might move
> them to WebKit's SVN.
> 

This is done now, as of:
https://trac.webkit.org/r261274

Philippe


> Philippe
> 
> [0] https://developer.gnome.org/jhbuild/stable/
> [1] https://trac.webkit.org/browser/trunk/Tools/gtk/jhbuild.modules
> [2] https://trac.webkit.org/browser/trunk/Tools/wpe/jhbuild.modules
> [3] https://bugs.webkit.org/show_bug.cgi?id=73425
> [4] https://flatpak.org/
> [5] https://github.com/endlessm/flapjack
> [6] https://bugs.webkit.org/show_bug.cgi?id=205658
> [7] https://flatpak.org/setup/
> [8] https://github.com/igalia/webkit-flatpak-sdk
> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>