[Webkit-unassigned] [Bug 69444] The new run-webkit-tests needs to dump out pixel hash failures even if the pixel test passes.

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=69444


Dirk Pranke <dpranke at chromium.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|webkit-unassigned at lists.web |dpranke at chromium.org
                   |kit.org                     |




--- Comment #6 from Dirk Pranke <dpranke at chromium.org>  2012-02-15 19:57:07 PST ---
Adding Ojan and Tony to this ... 

If memory serves, Chromium has always considered the case where the image hashes don't match but the images themselves do to be a passing case. I think we used to report a warning in the results.html file, but at some point we stopped doing that (either when we made the results file generated from the json data, or when we embedded the checksums into the image, I think).

The upshot is we currently ignore the problem, which seems wrong, but I'm not sure which is the right thing to do. I think we have four choices:

1) report a warning to the console but otherwise treat the test as passing across the board
2) report a warning to the console and include the warning in results.html / results.json, but consider the test as passing (similar to how we handle stderr output now)
3) consider the test as failing, but only if --tolerance is set to 0.0
4) consider the test as failing, regardless of what --tolerance is set to

In any situation, we'd probably need to change the results.html output to identify this case properly and not just confuse the user.

I have a patch that does #1 but only logs an error to the console that I will post, just to demonstrate some of the scope of work involved in fixing it.

I have no idea how often this will come up in practice so we probably don't want to overengineer this.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.