[Webkit-unassigned] [Bug 105860] Tests with WontFix expectation are (indirectly) skipped

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=105860





--- Comment #5 from Zan Dobersek <zandobersek at gmail.com>  2012-12-31 13:45:20 PST ---
(In reply to comment #3)
> Note that NRWT actually used to do what you wanted (i.e., WontFix did not imply Skip). There was a fair amount of debate over the years as to whether or not running the WontFix-but-not-Skip tests was at all useful for catching the sort of crashes you mention. The closest anyone came was the belief that it did catch a few things when the port was very new (i.e., we were still bringing it up and crashes were not uncommon).
> 
> Eventually we mostly agreed w/ Eric; the cost of running tests we didn't care about seemed excessive, especially given that we had so much other coverage, and so we changed what WontFix meant (IIRC, Apple also wanted WontFix to imply Skip, although they probably would've been happy w/ just Skip).

As said in comment #4, fast/harness/sample-fail-mismatch-reftest.html is an example of a layout test that should definitely be run but is not really fixable either. OTOH, it is probably the only test of such nature, but I urge every port to run it.

And again, the Chromium port marks ~20% of collected layout tests as WontFix. That's a good chunk of coverage out of the window right at the base of the complete Chromium project structure.

> But, not every port is Chromium - I could see an argument that other, especially newer, ports might want or need better coverage from the suite. I don't think we're likely to change back to the old behavior generally, but I could possibly be convinced to make this a port-specific configurable parameter whether things are skipped or not. Even then, this might be a bad idea since different ports would work differently.

I'd very like to see at least a compromise of this in the form of an opt-in configuration to run tests marked as WontFix.

I'd still prefer a possibility of an expanded modifier set, where the WontFix modifier would only indicate the non-fixable nature of the test while the extra modifier would indicate the actual expected failure. Using only the WontFix modifier could still just skip the test, as is done now.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.