[Webkit-unassigned] [Bug 66581] New: new-run-webkit-tests returns false-positives for chromium port if ImageDiff fails

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=66581

           Summary: new-run-webkit-tests returns false-positives for
                    chromium port if ImageDiff fails
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: UNCONFIRMED
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: shawnsingh at chromium.org
                CC: dpranke at chromium.org


if the ImageDiff tool fails, then the diff_image function (in ChromiumPort, Tools/Scripts/webkitpy/layout_tests/port/layoutchromium.py) returns "None" to indicate the test passed.  It does print an error in this case, but it is too easy to overlook it.

It is probably important to find a cleaner way to handle this.  Some options (thanks to dpranke):
 - the ideal option is to modify NRWT to handle internal errors so that tests are marked as "internal error, could not run test" instead of passed or failed.
 - modify the code to cause a test failure and outputs a diff, then the user can see that the diff is actually correct (false negatives may be safer than false positives)
 - optionally, have NRWT output a diff, with a hardcoded .png that literally says there was an internal error in the image.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.