[webkit-dev] insanity of updating 4000+ baseline images due to font rendering change?

[Elliot Poger]

We finally have correct baseline images checked in for Chromium-Skia on
SnowLeopard (hooray).  Now we face the daunting task of rebaselining for
Chromium-Skia on Leopard.

The problem is that chromium-mac-leopard has ~4500 image diffs compared to
chromium-mac.  The vast majority of these diffs are due to very minor
differences in text rendering between Leopard and SnowLeopard.  (Details are
in http://code.google.com/p/chromium/issues/detail?id=100904 )

I poked around in LayoutTests/platform (details are at the bottom of this
email) and found that indeed there are 4000-5000 baseline images that
distinguish between Leopard and SnowLeopard.  This is presumably why layout
tests generally pass on both platforms.

Before Cary and I spend 2-3 days committing all these new baseline images so
that we can re-enable test_expectations for SnowLeopard... is this the right
approach?  It seems pretty inefficient to clog the WebKit repository with
all these baseline images with minor pixel value differences.


Here are the various approaches I can think of... what's the
Hive-Mind-Approved approach?


   - Commit 4500 new baseline images for SnowLeopard
      - pro: known to work, will catch any regressions that come later
      - con: takes a long time to commit, chews up disk space and bandwidth
      for all developers, future minor changes may require yet another
set of new
      baselines
   - Leave all SnowLeopard tests marked as "PASS FAIL" (or maybe mark them
   "SKIP") in test_expectations
      - pro: known to work, quick and easy, doesn't clog repo space and
      developer update bandwidth, future minor changes won't break any bots
      - con: will not catch any regressions that come later on SnowLeopard
   - Remove descriptive text from all these tests, so that text rendering is
   only evaluated in tests specifically for that purpose
      - pro: prevents this problem for future OS versions, should allow for
      lots more baseline images to be shared across platforms
      - con: a lot of work to replace all existing baseline images, must
      coordinate across community of Chromium/WebKit developers, tests will be
      more difficult to interpret without text
   - Figure out how our test pages can be rendered with a completely
   cross-platform pixel-equivalent font
      - pro: similar to above but tests keep their descriptive text
      - con: similar to above but more technically challenging
   - Augment our pixel-diff tools to allow for comparison masks (only pay
   attention to pixel diffs within this rectangle)
      - pro: existing baseline images can stay in place, and perhaps be
      shared with new OS versions and platforms
      - con: requires modification of pixel-diff tools, need to add
      comparison mask to each test definition







Details on how I determined that there are already 4000-5000 baseline images
that distinguish between Leopard and SnowLeopard for existing platforms
(Safari and chromium-cg):

~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir tests
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ find
platform -name *.png >tests/all-pngs
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
tests/all-pngs
  45529 tests/all-pngs

~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/all-pngs | awk -F '/' '{print $2}' | sort | uniq >
tests/list-all-platforms
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ grep mac
tests/list-all-platforms >tests/list-mac-platforms

~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir
tests/by-platform
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ for PLATFORM
in $(cat tests/list-mac-platforms); do grep ^platform/$PLATFORM/
tests/all-pngs | sed s/platform\\/$PLATFORM\\///
>tests/by-platform/$PLATFORM ; done
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
tests/by-platform/*
   1050 tests/by-platform/chromium-cg-mac
   1575 tests/by-platform/chromium-cg-mac-leopard
    114 tests/by-platform/chromium-cg-mac-snowleopard
    114 tests/by-platform/chromium-gpu-cg-mac
    135 tests/by-platform/chromium-gpu-mac
   4909 tests/by-platform/chromium-mac
     79 tests/by-platform/chromium-mac-leopard
    131 tests/by-platform/chromium-mac-snowleopard
   7631 tests/by-platform/mac
   4435 tests/by-platform/mac-leopard
    247 tests/by-platform/mac-snowleopard
      7 tests/by-platform/mac-wk2
  20427 total

~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/by-platform/chromium-cg-mac
tests/by-platform/chromium-cg-mac-snowleopard tests/by-platform/mac
tests/by-platform/mac-snowleopard | sort | uniq
>tests/list-snowleopard-cg-images
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/by-platform/chromium-cg-mac-leopard
tests/by-platform/mac-leopard | sort | uniq
>tests/list-leopard-cg-images
~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
tests/list-leopard-cg-images tests/list-snowleopard-cg-images | sort |
uniq >tests/list-unique-cg-images

~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
tests/*-cg-images
   5120 tests/list-leopard-cg-images
   7735 tests/list-snowleopard-cg-images
   7744 tests/list-unique-cg-images
  20599 total
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111019/730d95e1/attachment.html>