[webkit-dev] run-webkit-tests question; hashes when comparing ref test output

[Darin Adler]

> On Jan 22, 2015, at 10:36 PM, Alexey Proskuryakov <ap at webkit.org> wrote:
> 
>> 22 янв. 2015 г., в 17:57, Darin Adler <darin at apple.com> написал(а):
>> 
>> What about the test I cited?
>> 
>> svg/css/svg-resource-fragment-identifier-img-src.html
> 
> This particular test is buggy - it is a hidpi test, so it dumps results as a 1600x1200 image, but its -expected.html is not hidpi, and is dumped as 800x600, so hashes are obviously different. I now filed <https://bugs.webkit.org/show_bug.cgi?id=140815> about this test.
> 
> When we compare pixels, we draw both images into bitmaps of the same size, so they become similar enough for ImageDiff to consider them identical.
> 
> Earlier today, Simon and I agreed that we should just silence the error message, because it only tells us about minor color differences that are inevitable when comparing compositing vs. non-compositing. Looks like it tells us about more actionable issues, too. Also, I just found <https://bugs.webkit.org/show_bug.cgi?id=69444>, and I think that its rationale applies in this case, too.
> 
> So we should probably keep this error message. I'm not sure whether we should make it a hard error though.

I think that to improve things we should make an informative message for this particular mistake where the expected file has the wrong size or resolution.

For me, the bad thing about the current message is not simply that it’s a false positive, but that it’s an unclear error message that covers too many different possibilities.

I’m not sure that keeping things as is will be a good strategy. A message that is often expected but sometimes indicates a real problem is not good for the project. The average engineer has no idea whether to ignore these or act on them!

— Darin