[Webkit-unassigned] [Bug 105860] Tests with WontFix expectation are (indirectly) skipped

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=105860





--- Comment #6 from Dirk Pranke <dpranke at chromium.org>  2013-01-01 09:19:32 PST ---
(In reply to comment #4)
> (In reply to comment #1)
> > I believe the extra crash-coverage we get from WONTFIX tests is so miniscule compared to the (also small) cost of maintaining any expectations for them, that it's not worth running them.
> 
> While I pointed out only the crash coverage in comment #0, there are other benefits. The fast/harness/sample-fail-mismatch-reftest.html[1][2] layout test tests that an image-only failure occurs when the reftest and its expected mismatch baseline provide equal output. This test is marked as WontFix on both Chromium and GTK (and probably other ports as well) but isn't run even though it's integral to checking the correct harness behavior on which plenty of tests rely. I remember one occurrence of problems in harness that were indicated by this test unexpectedly passing (but it wasn't reported due to an otherwise unrelated bug), so not running this test can backfire.
> 

That's a good counterexample.

> Also, the Chromium port currently marks over 6700 tests as WontFix. That's ~20% of all the tests the port collects. IMO that's way too much to ignore and not run.
> 

IIRC, a large chunk of these are the sputnik tests, which we don't run because v8 runs them as part of their own test suite. Otherwise most of the tests we mark WontFix are for features that are either port-specific or that we have no intention of implementing.

(In reply to comment #5)
> 
> I'd very like to see at least a compromise of this in the form of an opt-in configuration to run tests marked as WontFix.
> 
> I'd still prefer a possibility of an expanded modifier set, where the WontFix modifier would only indicate the non-fixable nature of the test while the extra modifier would indicate the actual expected failure. Using only the WontFix modifier could still just skip the test, as is done now.

That's an interesting suggestion; it's actually basically compatible with the existing syntax (since no modifiers implies Skip). I will take a look at that and if there's no real complications that that introduces, that seems like a reasonable thing to do.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.