<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div>I too am mildly concerned about references not being sufficiently independent of the tests, which is why I hoped we could get the WG in the business of reviewing references along with tests. However, another possibility is looking at what Mozilla uses for reference for these tests, since those would presumably have a track record of being good enough.<div><br></div><div>Regards,</div><div>Maciej<br><div><br><div><div>On Mar 7, 2012, at 4:58 PM, Ojan Vafai wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div class="gmail_quote">On Wed, Mar 7, 2012 at 4:50 PM, Ryosuke Niwa <span dir="ltr"><<a href="mailto:rniwa@webkit.org">rniwa@webkit.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div class="im">On Wed, Mar 7, 2012 at 4:44 PM, Darin Fisher <span dir="ltr"><<a href="mailto:darin@chromium.org" target="_blank">darin@chromium.org</a>></span> wrote:<br></div><div class="im">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hrm, if the test expectations are customized already for different ports of WebKit, then why not support replacing a PNG file with a HTML file that is intended to generate exactly the same result? How does this impair our ability to update the tests?<div>
<br></div><div>(I realize that our current reftest system may not work like this. I'm not familiar with the details of how it works in fact, but it seems like it could be as simple as having an expected result that is a HTML file instead of a PNG file.)</div>
</blockquote><div><br></div></div><div>How do we know that we are testing what the test is intending to test after the conversion? e.g. it's possible to create a reference file that fails to catch certain bugs.</div>
<div><br>
</div><div>It's not obvious to me how one would figure out how many reference files are needed for a given test to make sure we're not making the test more permissible than the author intended it to be.</div></div>
</blockquote><div><br></div><div>I'm not suggesting that we convert 100% of these tests to reftests, but a very very large percentage of them can easily be verified to be testing the correct thing and would only need a single reference file (it's just a line of text with a colored box in it after all). </div>
</div>
_______________________________________________<br>webkit-dev mailing list<br><a href="mailto:webkit-dev@lists.webkit.org">webkit-dev@lists.webkit.org</a><br>http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev<br></blockquote></div><br></div></div></body></html>