[Webkit-unassigned] [Bug 192391] New: Use hash in Unified Sources filename instead of ascending integers

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=192391

            Bug ID: 192391
           Summary: Use hash in Unified Sources filename instead of
                    ascending integers
           Product: WebKit
           Version: WebKit Nightly Build
          Hardware: Unspecified
                OS: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: New Bugs
          Assignee: webkit-unassigned at lists.webkit.org
          Reporter: cadubentzen at gmail.com
                CC: aperez at igalia.com, keith_miller at apple.com,
                    mcatanzaro at igalia.com

Created attachment 356571

  --> https://bugs.webkit.org/attachment.cgi?id=356571&action=review

Patch for the experiment

During a git-bisect session, I noticed that building times were taking too long even though I was:
- Using a compiler cache (CCache in case)
- Doing many WebKit builds in sequence

So I noticed that the building is stalled during UnifiedSources because the following may happen easily:
1) a new bundle (up to 8 source files) or more is added
2) all generated Unified Sources filenames (UnifiedSourceXYZ.cpp) after this bundle are shifted XYZ+N (N=number of new bundles)

This implies ccache having lots of cache misses.

To confirm that was the case, I did a little experiment (patch for the experiment attached):
- Cleared CCache and compiled WebKit from clean build directory

Time for a clean build (for comparison with build times below): 14973.56s user 1947.89s system 2514% cpu 11:12.97 total

- After, added an source file containing only `#include "config.h"` in a separate folder in JavaScriptCore, WebCore and WebKit e.g TestUnified/TestUnified.cpp inside these folders
- Added it to the beginning of each corresponding Sources.txt
  - It will cause a new bundle to be added because it only bundles together files in the same directory
  - The shifting will then occur
- Built WebKit again without cleaning the build directory.

Result:
- Time to build: 7315.90s user 727.49s system 2456% cpu 5:27.38 total

Half the time of clean-build with only 3 new "empty" source files.

To improve this build time, I changed the naming of UnifiedSources to UnifiedSources-{hash}.cpp where the hash is calculated based on the UnifiedSource text content (only the #include lines with the cpp files).

I then repeated the experiment:
- Time to build: 27.14s user 5.44s system 226% cpu 14.405 total

Just like building without changes.

Advantages:
- The Unified Source filename is based on its content rather than an arbitrary integer that may change frequently
- When new bundles are added, only that new bundle and the ones with sources modified are recompiled.

I think it should improve build time in the bots (if they do use compiler caching) and eventual git-bisect sessions in the future.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-unassigned/attachments/20181205/236c6ce0/attachment-0001.html>