[webkit-dev] Increasing the number of cross-platform/port expected results

[Robert O'Callahan]

> Mozilla has been using this technique for years. Perhaps we can pick their
> brains for some good tricks. Or, dare I say it, share some tests.
>
>
Hi, hope I'm not crashing the party, and sorry I'm late :-). Let me just say
a few things about reftests...

Maciej mentioned that a reftest can assert that <b> and
style="font-weight:bold" are equivalent, but would not catch a bug that
completely disabled font-weight. That is true. However, in our reftest
framework you can also assert that two pages render differently, so you can
test that font-weight:normal renders differently from font-weight:bold,
which would catch a bug like that. If you look in the Mozilla reftests,
there are many tests with 'sanity' in the name ensuring that pages that
should render differently, do.

You could still miss a weird bug like font-weight:bold making the text
italic instead. But in practice, we hardly ever encounter bugs that cause
some reftests to render incorrectly and still all tests pass.

Over time we have learned a number of tricks for writing reftests. Here are
a few:
-- In general we require tests to match pixel-perfectly. It's always
tempting to allow a fudge factor, but tiny differences can indicate
significant bugs.
-- We have nifty SVG filter tricks to help find those tiny differences, e.g.
http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-analyzer.xhtml
-- If some pixels of a test are unpredictable or platform-dependent, but not
relevant to the test, you can censor them out by placing an opaque element
over them in the test and reference.
-- In other tests with unpredictable pixel values, e.g. video, we use SVG
filters to threshold pixel channel values.
-- The Web usually has More Than One Way To Do It. E.g. for CSS gradients, a
lot of our reference pages use canvas to render a reference gradient.
-- Many Web features have particular cases that are easy to reduce to
reftests even when general cases aren't. For example, box-shadow and
text-shadow with no blur are trivial to test with reftests. You can create
gradients using repeated stops to achieve abrupt transitions and test them
against colored boxes. You could use a similar trick to test patterns in SVG
text.
-- Aggressive subpixel antialiasing is a real pain. For example, wrapping
text in an overflow:hidden container isn't always a no-op even if the
container is sized to its contents. Using sans-serif fonts helps, using CSS
padding helps more.
-- For speed, conciseness and overall expediency, you can often pack many
variations of a test into a single reftest page, and in practice there's no
significant downside to that.

One might argue that although reftests are a good way to test what they
test, they don't provide enough broad functional testing, e.g. to make sure
that blurs "look right". I don't have good data to refute or confirm that.
However, if we want to, we can get an image test just by making the
reference page a PNG ... but I can only see two cases out of over 4000 where
we felt we needed to do that.

Overall, I'm extremely enthusiastic about reftests! As others mentioned,
we'd like to get official CSS and SVG test suites using reftests; they're
the best approach I know of for writing robust automated layout and
rendering tests across platforms and browsers.

Rob
-- 
"He was pierced for our transgressions, he was crushed for our iniquities;
the punishment that brought us peace was upon him, and by his wounds we are
healed. We all, like sheep, have gone astray, each of us has turned to his
own way; and the LORD has laid on him the iniquity of us all." [Isaiah
53:5-6]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100309/7bf0b43a/attachment.html>