[webkit-dev] Pixel test differences between Leopard and Snow Leopard

[James Robinson]

Leopard and Snow Leopard have subtle differences in the way they render
antialiased text.  This does not affect the text metrics but does cause
slight pixel differences.  The majority of our existing pixel test baselines
appear to have been generated on Leopard, but there is a growing minority of
tests that have been generated on Snow Leopard as well as a small number of
baselines that are out of date or just plain incorrect.  This means that
when run with --tolerance=0, there are a significant number of failures on
Snow Leopard that do not have anything to do with the behavior they are
intended to test.  Here are some concrete numbers (taken around r59000, the
exact results vary slightly from revision to revision but not hugely).

On a Leopard machine, running the tests with -p --tolerance=0 yields 120
failures.  With the default tolerance (0.1%) there are 68 failures.

On Snow Leopard, with -p --tolerance=0 there are 3934 failures.  With the
default tolerance (0.1%) there are 144 failures.

Based on this, it seems reasonable to assume that there are ~3814 pixel test
results in platform/mac that are Leopard specific.  There is also a smaller
number of results generated from Snow Leopard machines that fail on Leopard.
 I think this is an unhealthy state for the project as it prevents people
developing on a Mac on Snow Leopard from running the pixel tests without
getting a huge number of spurious failures.  Currently pretty much all Mac
developers are likely to be on Snow Leopard, except for Google employees who
will be moving to Snow Leopard in the near future.  The pixel tests are our
only coverage for lots of the repaint logic currently.

I think fundamentally there is a short term and a longer term problem here.
 The short term problem is that since the Mac pixel results are in such a
bad state for people on Snow Leopard, people making changes to the code are
not likely to update the pixel results at all which leads to an even worse
mess in platform/mac.  The longer term problem is that since the pixel tests
are only run on build bots by one port (Chromium) and not run on
build.webkit.org by anyone, the pixel tests are likely to languish.

To resolve the immediate problem of Leopard-specific results in
platform/mac, I'd like to move all of the Leopard-specific results currently
in platform/mac to platform/mac-leopard and to generate new Snow Leopard
specific results for those tests in platform/mac.  Specifically my plan is
to do this:

For each test T with pixel results in platform/mac that fails on Leopard or
Snow Leopard with -p --tolerance=0:
- if T passes pixel tests exactly on Leopard and fails with a <0.5% pixel
difference on Snow Leopard, assume the only pixel difference is due to text
antialiasing and do the following:
 - svn mv the current platform/mac pixel results to platform/mac-leopard
 - add new pixel results generated on Snow Leopard into platform/mac
 - there should be around 3800 tests in this category
- if T passes pixel tests exactly on Snow Leopard and fails with a <0.5%
pixel difference on Leopard:
 - generate new pixel baselines for Leopard and add to platform/mac-leopard
 - there should be around 50 tests in this category
- if T fails on both Leopard and Snow Leopard, remove the current
expectations and file a bug
 - there should be around 70 tests in this category

After this is done it should be possible to run the full layout test suite
with --tolerance=0 on Snow Leopard and have no failures.  Assuming this
happens, I would like to switch the default value of --tolerance from 0.1 to
0.0 to try to keep regressions from creeping back in.

Does this sound like a good plan?  The biggest impact on the project will be
a number of large commits in LayoutTests/platform/ to do the svn move and
svn add operations to move results from platform/mac -> platform/mac-leopard
and to add new Snow Leopard results to platform/mac.  This can be done over
a weekend and mostly by a script to try to minimize impact.

- James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100517/0c6e659f/attachment.html>