[webkit-dev] Pixel test differences between Leopard and Snow Leopard

[Simon Fraser]

On May 17, 2010, at 3:44 PM, James Robinson wrote:

> Leopard and Snow Leopard have subtle differences in the way they render antialiased text.  This does not affect the text metrics but does cause slight pixel differences.  The majority of our existing pixel test baselines appear to have been generated on Leopard, but there is a growing minority of tests that have been generated on Snow Leopard as well as a small number of baselines that are out of date or just plain incorrect.  This means that when run with --tolerance=0, there are a significant number of failures on Snow Leopard that do not have anything to do with the behavior they are intended to test.  Here are some concrete numbers (taken around r59000, the exact results vary slightly from revision to revision but not hugely).
> 
> On a Leopard machine, running the tests with -p --tolerance=0 yields 120 failures.  With the default tolerance (0.1%) there are 68 failures.
> 
> On Snow Leopard, with -p --tolerance=0 there are 3934 failures.  With the default tolerance (0.1%) there are 144 failures.
> 
> Based on this, it seems reasonable to assume that there are ~3814 pixel test results in platform/mac that are Leopard specific.  There is also a smaller number of results generated from Snow Leopard machines that fail on Leopard.  I think this is an unhealthy state for the project as it prevents people developing on a Mac on Snow Leopard from running the pixel tests without getting a huge number of spurious failures.  Currently pretty much all Mac developers are likely to be on Snow Leopard, except for Google employees who will be moving to Snow Leopard in the near future.  The pixel tests are our only coverage for lots of the repaint logic currently.
> 
> I think fundamentally there is a short term and a longer term problem here.  The short term problem is that since the Mac pixel results are in such a bad state for people on Snow Leopard, people making changes to the code are not likely to update the pixel results at all which leads to an even worse mess in platform/mac.  The longer term problem is that since the pixel tests are only run on build bots by one port (Chromium) and not run on build.webkit.org by anyone, the pixel tests are likely to languish.

Thanks for doing this, James. At one point we had pixel bots at Apple, but they fell by the wayside.

> To resolve the immediate problem of Leopard-specific results in platform/mac, I'd like to move all of the Leopard-specific results currently in platform/mac to platform/mac-leopard and to generate new Snow Leopard specific results for those tests in platform/mac.  Specifically my plan is to do this:
> 
> For each test T with pixel results in platform/mac that fails on Leopard or Snow Leopard with -p --tolerance=0:
> - if T passes pixel tests exactly on Leopard and fails with a <0.5% pixel difference on Snow Leopard, assume the only pixel difference is due to text antialiasing and do the following:
>  - svn mv the current platform/mac pixel results to platform/mac-leopard
>  - add new pixel results generated on Snow Leopard into platform/mac
>  - there should be around 3800 tests in this category
> - if T passes pixel tests exactly on Snow Leopard and fails with a <0.5% pixel difference on Leopard:
>  - generate new pixel baselines for Leopard and add to platform/mac-leopard
>  - there should be around 50 tests in this category
> - if T fails on both Leopard and Snow Leopard, remove the current expectations and file a bug
>  - there should be around 70 tests in this category
> 
> After this is done it should be possible to run the full layout test suite with --tolerance=0 on Snow Leopard and have no failures.  Assuming this happens, I would like to switch the default value of --tolerance from 0.1 to 0.0 to try to keep regressions from creeping back in.

You may find that some of the text antialiasing differences are hardware-dependent, so I'm not sure if a tolerance of 0 would be possible.

> Does this sound like a good plan?  The biggest impact on the project will be a number of large commits in LayoutTests/platform/ to do the svn move and svn add operations to move results from platform/mac -> platform/mac-leopard and to add new Snow Leopard results to platform/mac.  This can be done over a weekend and mostly by a script to try to minimize impact.

This sounds like a great short-term plan. I think a longer-term plan would be to move as many tests to ref-tests as possible, which eliminates any text-aliasing issues.

Simon



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100517/5f8ac27c/attachment.html>