[webkit-changes] [WebKit/WebKit] 4b85cb: [cmake] Auto-complete via clangd auto-setup v2

Tue Aug 13 11:21:29 PDT 2024

  Branch: refs/heads/main
  Home:   https://github.com/WebKit/WebKit
  Commit: 4b85cbe26eff585fe8586af43bd7199f326b0caf
      https://github.com/WebKit/WebKit/commit/4b85cbe26eff585fe8586af43bd7199f326b0caf
  Author: Alicia Boya Garcia <aboya at igalia.com>
  Date:   2024-08-13 (Tue, 13 Aug 2024)

  Changed paths:
    R .clangd
    M .gitignore
    M Source/WTF/WTF.xcodeproj/project.pbxproj
    M Source/cmake/WebKitCommon.cmake
    A Tools/clangd/clangd auto-setup.png
    A Tools/clangd/clangd auto-setup.svg
    A Tools/clangd/clangd-check-codebase
    A Tools/clangd/clangd-check-file
    A Tools/clangd/clangd-config.yaml.tpl
    A Tools/clangd/update-clangd-config
    A Tools/clangd/update-compile-commands-symlink
    A Tools/clangd/update-compile-commands-symlink-conf
    A Tools/clangd/update-compile-commands-symlink.conf.example

  Log Message:
  -----------
  [cmake] Auto-complete via clangd auto-setup v2
https://bugs.webkit.org/show_bug.cgi?id=274909

Reviewed by Elliott Williams.

Re-upload of the patch including a fix for the internal Apple builds
that do not contain a Tools/ directory. See
https://bugs.webkit.org/show_bug.cgi?id=274909#c7 for details.

Additionally, this version has been modified to not depend on PyYAML or
Jinja2 and use simpler stdlib replacements instead, as Apple internal
builds cannot use any non-builtin Python packages.

---

 # Goals

This patch aims to bring out-of-the box completion with clangd for
WebKit developers using CMake builds.

clangd is an open source language server for providing code-completion,
go-to-definition and similar smart features for C++ code by analyzing it
with the same backend clang uses. **You do not need to compile WebKit
with clang to take advantage of this**, as clang strives to be flag
compatible with GCC, and even MSVC through `clang-cl`.

Using clangd with WebKit was already possible before, but required
manual intervention in the project files to set it up.

This patch attempts to clear some of the existing pitfalls,
particularly for CMake builds, so that more people take advantage of
Clangd without having to put conscious effort into troubleshooting.

This patch also makes WebKit stock clangd setup compatible with both
Mac and non-Mac platforms and allows developers to customize clangd
configuration without dirtying their repository.

---

Getting clangd right takes a lot of context and moving pieces, so this
is one of those very long commit messages. Sections with (NEW) in the
title explain changes introduced by this patch, whereas the rest hope to
explain the background and why those changes are necessary or useful.

 # Architecture (NEW)

Several scripts have been added in Tools/clangd. A diagram describing
the build system targets, dependencies and products is also included.

The clangd auto-setup needs to generate the following artifacts:
 * .clangd configuration file
 * compile_commands.json file/symlink in the WebKit repo root.

Starting with this patch `.clangd` is no longer committed in the
repository it is now .gitignore'd and auto-generated during the build.

 # Compatibility with XCode builds (NEW)

This patch doesn't improve or worsen the status of clangd support in
XCode builds.

XCode build users are still expected to generate compile_commands.json
manually as per the wiki states:
https://trac.webkit.org/wiki/Clangd#macOS1

    make release EXPORT_COMPILE_COMMANDS=YES
    generate-compile-commands WebKitBuild/Release

The generation of the `.clangd` file is also included in the XCode
build, as part of the WTF XCode project.

 # clangd config generation (NEW)

The script `update-clangd-config` autogenerates a `.clangd` file with an
«autogenerated» marker at the top. The file will only be updated as long
as that marker exists, therefore allowing end-user customization if
needed.

The `.clangd` file is generated from a template, which will produce
slightly different options for Mac and non-Mac platforms, fixing the
regression introduced in https://github.com/WebKit/WebKit/pull/23273.
The template previously used Jinja2, but in this version of the patch
uses the more rudimentary stdlib's strings.Template to avoid depending
on external libraries.

`update-clangd-config` is made part of the build by default. For XCode
builds, it is generated as part of WTF. For CMake builds, it is enabled
as a default target as long as CLANGD_AUTO_SETUP is set, which is the
default when building inside the WebKitBuild directory.

 # clangd configuration changes (NEW)

This patch also makes several changes to the clangd configuration file
to fix issues detected during testing:

 * UnifiedSource files are no longer indexed.
 * The `-fcoroutines` flag is stripped, as it is not recognized by some
   versions of clang, and by extension, clangd.
 * `-xobjective-c++` is now only passed in macOS to avoid depending on
   macOS-only headers in other systems.
 * `config.h` is no longer prepended to files within the ThirdParty
   directory.

 # compile_commands.json generation: context

CMake already generates a compile_commands.json file when the Ninja or
Makefile file is generated as long as CMAKE_EXPORT_COMPILE_COMMANDS is
ON, which is already the default in DEVELOPER_MODE.

When using unified builds (which is the default), that
`compile_commands.json` is somewhat subpar, as the language server is
unaware of most of the source .cpp files and hence has to require to
heuristics to match them to other build commands.

On the CMake side, a RewriteCompileCommands target exists that parses
that file along with unified sources to generate an improved file at
`<BUILD_DIR>/DeveloperTools/compile_commands.json`. Unfortunately, this
may not be obvious to many developers, which may be unaware of its
existence.

On the XCode builds side, this responsibility would fall under the
`generate-compile-commands` script, but it's seemingly not currently
implemented. Fixing that shouldn't be a difficult change but it's
outside of the scope of this patch. For more details:
https://github.com/WebKit/WebKit/pull/30784#issuecomment-2264933253

 # Same compile_commands.json path for non-unified builds (NEW)

The first change this patch makes is solving the unified builds vs
non-unified builds dichotomy by introducing a new build target for
non-unified builds named SymlinkCompileCommandsInDeveloperTools.

This way, **no matter the type of build**,
`<BUILD_DIR>/DeveloperTools/compile_commands.json` is the path to be
relied upon.

 # compile_commands.json vs builds

For a `compile_commands.json` file to be used, it needs to exist in a
parent directory of the sources being worked on.

Since any generated `compile_commands.json` is specific for a given build
(e.g. Debug vs Release), with build-specific flags, developers need to
make a choice of what build to use for the `compile_commands.json` in
their WebKit checkout.

In XCode builds, this is passed explicitly to the
generate-compile-commands command.

In CMake builds, this is usually accomplished by symlinking the file
into the root of the WebKit repository.

Developers who frequently switch and remove builds need to be careful to
not let the compile_commands.json symlink become broken, as that would
cause the language server to rely on much worsened heuristics.

 # compile_commands.json symlink autogeneration and management (NEW)

This patch introduces automatic compile_commands.json symlink handling.
A new target is introduced that intentionally runs with every build:
`UpdateCompileCommandsSymlink`, which relies on the
`update-compile-commands-symlink` script, also introduced by this patch.
The script is meant to be fast enough to not introduce any perceivable
delay in builds.

`update-compile-commands-symlink` receives as arguments a symlink path
and a configuration file listing a series of candidate paths. The
symlink is updated to point to the first candidate path that exists.
In the previous version of this patch, that file was in YAML format, but
this version uses Python's ConfigParse INI syntax to avoid depending on
any external libraries.

This list of candidate paths is declared in a new .gitignore'd
`update-compile-commands-symlink.conf` file that is generated early
during any CMake build that uses `CLANGD_AUTO_SETUP`.

`update-compile-commands-symlink.conf` will be automatically updated
from `Tools/clangd/update-compile-commands-symlink.conf.example`; but
similarly to `.clangd`, a user can disable this and manage the file on
their own, in this case by setting the `user_managed` property inside
the file to True.

 # Testing

clangd auto-setup has been tested with a Linux x86_64 machine using
webkit-container-sdk and Visual Studio Code, both with the default GCC
and with clang 16, with a fresh installation of VS Code (no settings).

It is possible to confirm if a file passes under clangd by using the
`clangd --check` command along with the compile lines used during
compilation, which can be queried from compile_commands.json.

A couple barebones scripts showing this usage have been provided in
Tools/clangd as documentation: `clangd-check-file` and
`clangd-check-codebase`.

I was able to run clangd-check-codebase with very few .cpp files
failing, and the failing files had good reasons for failing, like
missing includes that were obscured by unified builds.

Canonical link: https://commits.webkit.org/282175@main

To unsubscribe from these emails, change your notification settings at https://github.com/WebKit/WebKit/settings/notifications