[webkit-dev] Increasing the number of cross-platform/port expected results

[Jeremy Orlow]

On Tue, Mar 9, 2010 at 9:19 AM, Robert O'Callahan <robert at ocallahan.org>wrote:

>
> Mozilla has been using this technique for years. Perhaps we can pick their
>> brains for some good tricks. Or, dare I say it, share some tests.
>>
>>
>>
> Hi, hope I'm not crashing the party, and sorry I'm late :-). Let me just
> say a few things about reftests...
>
> Maciej mentioned that a reftest can assert that <b> and
> style="font-weight:bold" are equivalent, but would not catch a bug that
> completely disabled font-weight. That is true. However, in our reftest
> framework you can also assert that two pages render differently, so you can
> test that font-weight:normal renders differently from font-weight:bold,
> which would catch a bug like that. If you look in the Mozilla reftests,
> there are many tests with 'sanity' in the name ensuring that pages that
> should render differently, do.
>
> You could still miss a weird bug like font-weight:bold making the text
> italic instead. But in practice, we hardly ever encounter bugs that cause
> some reftests to render incorrectly and still all tests pass.
>
> Over time we have learned a number of tricks for writing reftests. Here are
> a few:
> -- In general we require tests to match pixel-perfectly. It's always
> tempting to allow a fudge factor, but tiny differences can indicate
> significant bugs.
> -- We have nifty SVG filter tricks to help find those tiny differences,
> e.g.
> http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-analyzer.xhtml
> -- If some pixels of a test are unpredictable or platform-dependent, but
> not relevant to the test, you can censor them out by placing an opaque
> element over them in the test and reference.
>

This and several of the other tips in here seem like they can apply nicely
to our existing layout tests as well.


> -- In other tests with unpredictable pixel values, e.g. video, we use SVG
> filters to threshold pixel channel values.
> -- The Web usually has More Than One Way To Do It. E.g. for CSS gradients,
> a lot of our reference pages use canvas to render a reference gradient.
> -- Many Web features have particular cases that are easy to reduce to
> reftests even when general cases aren't. For example, box-shadow and
> text-shadow with no blur are trivial to test with reftests. You can create
> gradients using repeated stops to achieve abrupt transitions and test them
> against colored boxes. You could use a similar trick to test patterns in SVG
> text.
> -- Aggressive subpixel antialiasing is a real pain. For example, wrapping
> text in an overflow:hidden container isn't always a no-op even if the
> container is sized to its contents. Using sans-serif fonts helps, using CSS
> padding helps more.
> -- For speed, conciseness and overall expediency, you can often pack many
> variations of a test into a single reftest page, and in practice there's no
> significant downside to that.
>
> One might argue that although reftests are a good way to test what they
> test, they don't provide enough broad functional testing, e.g. to make sure
> that blurs "look right". I don't have good data to refute or confirm that.
> However, if we want to, we can get an image test just by making the
> reference page a PNG ... but I can only see two cases out of over 4000 where
> we felt we needed to do that.
>
> Overall, I'm extremely enthusiastic about reftests! As others mentioned,
> we'd like to get official CSS and SVG test suites using reftests; they're
> the best approach I know of for writing robust automated layout and
> rendering tests across platforms and browsers.
>
> Rob
> --
> "He was pierced for our transgressions, he was crushed for our iniquities;
> the punishment that brought us peace was upon him, and by his wounds we are
> healed. We all, like sheep, have gone astray, each of us has turned to his
> own way; and the LORD has laid on him the iniquity of us all." [Isaiah
> 53:5-6]
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100309/de5a2c8d/attachment.html>