[webkit-dev] Increasing the number of cross-platform/port expected results

[Tor Arne Vestbø]

Hey all,

A reoccurring problem when trying to maintain layout-test results is 
differences in font and theme metrics for tests that dump the render 
tree. Often a test does not actually test font loading/rendering or 
theming, but has a piece of text or an input element somewhere in the 
test which causes the metrics in the render tree to be different, and 
hence the test failing.

We're seeing this in the Qt port when results are checked in as 
platform-independent and causing failures, or when we have to generate 
platform-dependent results for new tests when the test looks cross-platform.

I also had this problem when trying to get a Windows build up and 
running with 0 failing layout-tests, so that I could verify that I 
didn't break anything before landing. Even after using various tricks to 
get the same font setup as the buildbots I still had failures that 
looked like they were due to font metrics [1].

Lately we've been playing with the idea of using SVG fonts for the Qt 
port to get the same set of expected results for qt-mac, qt-linux and 
qt-win, by injecting new @font-face rules using a user-stylesheet and 
preventing platform-fonts from being loaded, but this approach/hack has 
proven to be quite fragile, and we will also miss out on those tests 
that actually test font loading/rendering.

Another approach would be to use a fixed set of metrics for text and 
themes when running the DRT, unless the test specifically asks for real 
font and theme metrics. We could for example add two additional 
functions to the layout-test controller:

    layoutTestController.dumpRealFontMetrics()

and

    layoutTestController.dumpRealThemeMetrics()

This would allow us to share render-tree results between all ports for 
those tests that don't explicitly test font or theme rendering (unless I 
missed more platform-differences).

Thoughts? Will this work? :)

Or, are there other ways we could achieve the same thing?

My worry is basically that the number of platform-specific-result will 
not scale as the number of ports and platforms increases if we continue 
to have all these "false positive" platform-specific-results that are 
not really platform-specific.

Tor Arne

[1] 
http://trac.webkit.org/wiki/BuildingOnWindows#RunningtheLayoutTestsonWindows