[webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Levi Weintraub]

On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher <darin at chromium.org> wrote:
>
>> Hrm, if the test expectations are customized already for different ports
>> of WebKit, then why not support replacing a PNG file with a HTML file that
>> is intended to generate exactly the same result?  How does this impair our
>> ability to update the tests?
>>
>> (I realize that our current reftest system may not work like this.  I'm
>> not familiar with the details of how it works in fact, but it seems like it
>> could be as simple as having an expected result that is a HTML file instead
>> of a PNG file.)
>>
>
> How do we know that we are testing what the test is intending to test
> after the conversion? e.g. it's possible to create a reference file that
> fails to catch certain bugs.
>

This sums up my worry as well. I can imagine a bug causing a CSS test and
its reference to fail in the same way, masking the failure.


>
> It's not obvious to me how one would figure out how many reference files
> are needed for a given test to make sure we're not making the test more
> permissible than the author intended it to be.
>
> - Ryosuke
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120307/cab3bdf9/attachment.html>