[Webkit-unassigned] [Bug 80644] Test expectations and tolerance interfere

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=80644





--- Comment #3 from Dirk Pranke <dpranke at chromium.org>  2012-03-09 13:40:32 PST ---
(In reply to comment #2)
> (In reply to comment #1)
> > I don't think I understand what you're looking for ... the difference between an expected failure and an unexpected failure is that the former don't cause the test run itself to fail (since the tree goes red only on unexpected failures).
> > 
> > In your example, do you want css3/filters/crash-filter-change.html to turn the tree red if the pixel hash doesn't match? If so, why list it as an expected IMAGE failure at all?
> > 
> > Perhaps we need some sort of expectation of fuzzy pass but exact failure?
> 
> Yes, I think that's what we want. A fuzzy pass expectation (the overall result may be a pass because the tolerance can change, but the pixel hash will not pass).

Okay, so is it that you don't want the "unexpected pass" line to show up, and to just see the pixel hash failed/pixel test passes line?

Or do you pixel hash failures to show up in results.html as well?

Or do you want the tests to actually fail and turn the step red?

Either of the first two is fine with me, I'm just trying to understand what to implement. if you want the third behavior, I'm still confused.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.