[webkit-dev] insanity of updating 4000+ baseline images due to font rendering change?

[Adam Barth]

I definitely agree that our current approach to pixel baselines is
less than ideal.  There have been a number of efforts to reduce the
number of pixel tests, for example by converting tests to text tests
or to ref tests, but even after those efforts, we still have thousands
of tests that require different baselines on essentially every
platform and port.

We'd prefer not to leave the Chromium Leopard test expectations has
"PASS FAIL" in the long term because that removes a large amount of
test coverage (~20%).

How confident are you that flipping the CG -> Skia switch will stick this time?

Once we're sure the switch will stay in the Skia position, we can
delete the Chromium Leopard CG expected results, which will leave the
net quantity of results roughly the same.  Also, once you're confident
in the Leopard Skia results, there shouldn't be too much churn (e.g.,
as there was when fixing the focus ring and the tiny font rendering).

Adam


On Wed, Oct 19, 2011 at 2:04 PM, Elliot Poger <epoger at google.com> wrote:
> We finally have correct baseline images checked in for Chromium-Skia on
> SnowLeopard (hooray).  Now we face the daunting task of rebaselining for
> Chromium-Skia on Leopard.
> The problem is that chromium-mac-leopard has ~4500 image diffs compared to
> chromium-mac.  The vast majority of these diffs are due to very minor
> differences in text rendering between Leopard and SnowLeopard.  (Details are
> in http://code.google.com/p/chromium/issues/detail?id=100904 )
> I poked around in LayoutTests/platform (details are at the bottom of this
> email) and found that indeed there are 4000-5000 baseline images that
> distinguish between Leopard and SnowLeopard.  This is presumably why layout
> tests generally pass on both platforms.
> Before Cary and I spend 2-3 days committing all these new baseline images so
> that we can re-enable test_expectations for SnowLeopard... is this the right
> approach?  It seems pretty inefficient to clog the WebKit repository with
> all these baseline images with minor pixel value differences.
>
> Here are the various approaches I can think of... what's the
> Hive-Mind-Approved approach?
>
> Commit 4500 new baseline images for SnowLeopard
>
> pro: known to work, will catch any regressions that come later
> con: takes a long time to commit, chews up disk space and bandwidth for all
> developers, future minor changes may require yet another set of new
> baselines
>
> Leave all SnowLeopard tests marked as "PASS FAIL" (or maybe mark them
> "SKIP") in test_expectations
>
> pro: known to work, quick and easy, doesn't clog repo space and developer
> update bandwidth, future minor changes won't break any bots
> con: will not catch any regressions that come later on SnowLeopard
>
> Remove descriptive text from all these tests, so that text rendering is only
> evaluated in tests specifically for that purpose
>
> pro: prevents this problem for future OS versions, should allow for lots
> more baseline images to be shared across platforms
> con: a lot of work to replace all existing baseline images, must coordinate
> across community of Chromium/WebKit developers, tests will be more difficult
> to interpret without text
>
> Figure out how our test pages can be rendered with a completely
> cross-platform pixel-equivalent font
>
> pro: similar to above but tests keep their descriptive text
> con: similar to above but more technically challenging
>
> Augment our pixel-diff tools to allow for comparison masks (only pay
> attention to pixel diffs within this rectangle)
>
> pro: existing baseline images can stay in place, and perhaps be shared with
> new OS versions and platforms
> con: requires modification of pixel-diff tools, need to add comparison mask
> to each test definition
>
>
>
>
>
> Details on how I determined that there are already 4000-5000 baseline images
> that distinguish between Leopard and SnowLeopard for existing platforms
> (Safari and chromium-cg):
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir tests
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ find
> platform -name *.png >tests/all-pngs
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
> tests/all-pngs
>   45529 tests/all-pngs
>
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
> tests/all-pngs | awk -F '/' '{print $2}' | sort | uniq >
> tests/list-all-platforms
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ grep mac
> tests/list-all-platforms >tests/list-mac-platforms
>
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ mkdir
> tests/by-platform
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ for PLATFORM
> in $(cat tests/list-mac-platforms); do grep ^platform/$PLATFORM/
> tests/all-pngs | sed s/platform\\/$PLATFORM\\///
>>tests/by-platform/$PLATFORM ; done
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
> tests/by-platform/*
>    1050 tests/by-platform/chromium-cg-mac
>    1575 tests/by-platform/chromium-cg-mac-leopard
>     114 tests/by-platform/chromium-cg-mac-snowleopard
>     114 tests/by-platform/chromium-gpu-cg-mac
>     135 tests/by-platform/chromium-gpu-mac
>    4909 tests/by-platform/chromium-mac
>      79 tests/by-platform/chromium-mac-leopard
>     131 tests/by-platform/chromium-mac-snowleopard
>    7631 tests/by-platform/mac
>    4435 tests/by-platform/mac-leopard
>     247 tests/by-platform/mac-snowleopard
>       7 tests/by-platform/mac-wk2
>   20427 total
>
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
> tests/by-platform/chromium-cg-mac
> tests/by-platform/chromium-cg-mac-snowleopard tests/by-platform/mac
> tests/by-platform/mac-snowleopard | sort | uniq
>>tests/list-snowleopard-cg-images
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
> tests/by-platform/chromium-cg-mac-leopard
> tests/by-platform/mac-leopard | sort | uniq
>>tests/list-leopard-cg-images
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ cat
> tests/list-leopard-cg-images tests/list-snowleopard-cg-images | sort |
> uniq >tests/list-unique-cg-images
>
> ~/src/webkit/rebaseline-greenify-bots/WebKit/LayoutTests$ wc -l
> tests/*-cg-images
>    5120 tests/list-leopard-cg-images
>    7735 tests/list-snowleopard-cg-images
>    7744 tests/list-unique-cg-images
>   20599 total
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>