[Webkit-unassigned] [Bug 80644] Test expectations and tolerance interfere

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=80644





--- Comment #1 from Dirk Pranke <dpranke at chromium.org>  2012-03-09 12:00:39 PST ---
(In reply to comment #0)
> Scenario:
> 
> Have some pixel tests which pass at the default tolerance, but fail at tolerance=0.
> 
> Run tests with tolerance=0, add the failing tests to text_expectations.txt as IMAGE failures.
> 
> Run tests with default tolerance, and the tests result in:
> 
> css3/filters/crash-filter-change.html -> pixel hash failed (but pixel test still passes)
> css3/filters/crash-filter-change.html -> unexpected pass
> 
> This is kind of unfortunate. I'm not sure how one would solve it, though. Perhaps if pixel hash fails and it's an expected failure, don't PASS it?

I don't think I understand what you're looking for ... the difference between an expected failure and an unexpected failure is that the former don't cause the test run itself to fail (since the tree goes red only on unexpected failures).

In your example, do you want css3/filters/crash-filter-change.html to turn the tree red if the pixel hash doesn't match? If so, why list it as an expected IMAGE failure at all?

Perhaps we need some sort of expectation of fuzzy pass but exact failure?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.