[Webkit-unassigned] [Bug 58804] New: new-run-webkit-tests doesn't correctly handle image failures below tolerance

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=58804

           Summary: new-run-webkit-tests doesn't correctly handle image
                    failures below tolerance
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: PC
        OS/Version: Mac OS X 10.5
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: ojan at chromium.org
                CC: eric at webkit.org, dpranke at chromium.org


I had an image failure for the snowleopard mac port (i.e. not --chromium). The image diff was below the default 0.1 tolerance. Here's what I got:
new-run-webkit-tests --experimental-fully-parallel --child-processes 10 --no-show-results --time-out-ms 1 --no-retry-failures fullscreen -p
  fullscreen/full-screen-zIndex.html -> unexpected image and text mismatch
9 tests ran as expected, 1 didn't:                 

Regressions: Unexpected image and text mismatch : (1)
  fullscreen/full-screen-zIndex.html = IMAGE+TEXT


What I should have gotten is something like the following:
new-run-webkit-tests --experimental-fully-parallel --child-processes 10 --no-show-results --time-out-ms 1 --no-retry-failures fullscreen -p
  fullscreen/full-screen-zIndex.html -> unexpected text mismatch and image hash failure (below 0.1% tolerance)
9 tests ran as expected, 1 didn't:                 

Regressions: Unexpected text mismatch : (1)
  fullscreen/full-screen-zIndex.html =TEXT


2 things went wrong:
-Should show an image failure below tolerance
-Should not count as an IMAGE+TEXT, but just a TEXT

Also, it didn't write out a -diff.png file. I'm not sure whether we should or not. We should do whatever old-run-webkit-tests did in this case.

When I run the above with tolerance=0, everything works out as expected (e.g. we print the same thing and we write out a -diff.png file).

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.