[webkit-dev] Increasing the number of cross-platform/port expected results

[Darin Adler]

On Feb 23, 2010, at 11:06 AM, Andras Becsi wrote:

> I like this concept of reftests because it seems easier to maintain and scales better on multiple ports than the current form of testing.

A reftest-style test seems like a good alternative to a pure text test. It can handle testing a lot more things but has a similar level of cross-platform portability.

It’s great that reftest-style tests are not as dependent on font metrics as render tree dumps and not as dependent on rendering differences as pixel tests are. But depending on the reftest machinery is implemented these may only be good for simple tests that fit on one screen, a limitation that currently affects only pixel test results, and these kinds of tests can miss breakages that pixel tests or render tree dumps could catch.

If we convert too many tests we’ll lose coverage. It’s probably common for both the reference rendering and the test case to both render wrong for the same reason. And in quite a few cases we can detect this type of regression even with just a render tree dump.

Today we have two types of tests:

    1) Plain text result only.
    2) Render tree test plus pixel result.

But in practice pixel results are often ignored entirely. I think that reftest-style tests if done right could be a great addition.

    -- Darin

PS: Someone, please, rename DumpRenderTree! ;-)