[webkit-dev] run-webkit-tests question; hashes when comparing ref test output

[Simon Fraser]

> On Jan 22, 2015, at 8:44 AM, Darin Adler <darin at apple.com> wrote:
> 
> I noticed that when running tests I get messages like this:
> 
>    svg/css/svg-resource-fragment-identifier-img-src.html -> ref test hashes didn't match but diff passed
> 
> I understand that we can have hash mismatches in actual checked in images (expected.png files), and there, at least, I believe the fix might be to regenerate the image. But for a ref test I am really unclear on this. And further, I think it’s distracting to get messages like this unless you are specifically working on the testing infrastructure.
> 
> Why should the average WebKit engineer care about this? Does this reflect a real performance problem when running tests? If so, who would fix it and what would that person do?

Alexey, Tim and I looked at this yesterday.

This happens when the expected and actual images are very close, but not identical. ImageDiff has some built-in rounding that effectively acts as a small tolerance, so the hashes are different, but ImageDiff incorrectly says the images are the same. For example, some of the tests in question render a green box either via CALayers, or by painting, and there’s a slight color difference between the two code paths.

My preference for how to fix this would be to fix ImageDiff to remove its slight built-in tolerance, and then to expose testRunner API to allow a test to set an explicit tolerance. There are many cases where we’d like to use ref tests, but are unable to because of slight, justifiable rendering differences, and having an explicit tolerance would permit the use of ref tests in these cases.

Simon