[Webkit-unassigned] [Bug 80644] New: Test expectations and tolerance interfere

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=80644

           Summary: Test expectations and tolerance interfere
           Product: WebKit
           Version: 528+ (Nightly build)
          Platform: Unspecified
        OS/Version: Unspecified
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: Tools / Tests
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: timothy_horton at apple.com
                CC: simon.fraser at apple.com


Scenario:

Have some pixel tests which pass at the default tolerance, but fail at tolerance=0.

Run tests with tolerance=0, add the failing tests to text_expectations.txt as IMAGE failures.

Run tests with default tolerance, and the tests result in:

css3/filters/crash-filter-change.html -> pixel hash failed (but pixel test still passes)
css3/filters/crash-filter-change.html -> unexpected pass

This is kind of unfortunate. I'm not sure how one would solve it, though. Perhaps if pixel hash fails and it's an expected failure, don't PASS it?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.