[webkit-dev] Is converting pixel tests to reftests kosher for imported libraries?

[Maciej Stachowiak]

I too am mildly concerned about references not being sufficiently independent of the tests, which is why I hoped we could get the WG in the business of reviewing references along with tests. However, another possibility is looking at what Mozilla uses for reference for these tests, since those would presumably have a track record of being good enough.

Regards,
Maciej

On Mar 7, 2012, at 4:58 PM, Ojan Vafai wrote:

> On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher <darin at chromium.org> wrote:
> Hrm, if the test expectations are customized already for different ports of WebKit, then why not support replacing a PNG file with a HTML file that is intended to generate exactly the same result?  How does this impair our ability to update the tests?
> 
> (I realize that our current reftest system may not work like this.  I'm not familiar with the details of how it works in fact, but it seems like it could be as simple as having an expected result that is a HTML file instead of a PNG file.)
> 
> How do we know that we are testing what the test is intending to test after the conversion? e.g. it's possible to create a reference file that fails to catch certain bugs.
> 
> It's not obvious to me how one would figure out how many reference files are needed for a given test to make sure we're not making the test more permissible than the author intended it to be.
> 
> I'm not suggesting that we convert 100% of these tests to reftests, but a very very large percentage of them can easily be verified to be testing the correct thing and would only need a single reference file (it's just a line of text with a colored box in it after all). 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120307/aaff2990/attachment.html>