[webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Tor Arne Vestbø]

On 08.03.12 01:57, Levi Weintraub wrote:
> On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa <rniwa at webkit.org
> <mailto:rniwa at webkit.org>> wrote:
>
>     On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher <darin at chromium.org
>     <mailto:darin at chromium.org>> wrote:
>
>         Hrm, if the test expectations are customized already for
>         different ports of WebKit, then why not support replacing a PNG
>         file with a HTML file that is intended to generate exactly the
>         same result?  How does this impair our ability to update the tests?
>
>         (I realize that our current reftest system may not work like
>         this.  I'm not familiar with the details of how it works in
>         fact, but it seems like it could be as simple as having an
>         expected result that is a HTML file instead of a PNG file.)
>
>
>     How do we know that we are testing what the test is intending to
>     test after the conversion? e.g. it's possible to create a reference
>     file that fails to catch certain bugs.
>
>
> This sums up my worry as well. I can imagine a bug causing a CSS test
> and its reference to fail in the same way, masking the failure.

What if the reference is one line of text + image that represents the 
expected result? That way ports can share the associated png.

tor arne