[Webkit-unassigned] [Bug 105860] Tests with WontFix expectation are (indirectly) skipped

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=105860





--- Comment #4 from Zan Dobersek <zandobersek at gmail.com>  2012-12-31 13:30:26 PST ---
(In reply to comment #1)
> I believe the extra crash-coverage we get from WONTFIX tests is so miniscule compared to the (also small) cost of maintaining any expectations for them, that it's not worth running them.

While I pointed out only the crash coverage in comment #0, there are other benefits. The fast/harness/sample-fail-mismatch-reftest.html[1][2] layout test tests that an image-only failure occurs when the reftest and its expected mismatch baseline provide equal output. This test is marked as WontFix on both Chromium and GTK (and probably other ports as well) but isn't run even though it's integral to checking the correct harness behavior on which plenty of tests rely. I remember one occurrence of problems in harness that were indicated by this test unexpectedly passing (but it wasn't reported due to an otherwise unrelated bug), so not running this test can backfire.

Also, the Chromium port currently marks over 6700 tests as WontFix. That's ~20% of all the tests the port collects. IMO that's way too much to ignore and not run.

> NRWT has various options to allow running skipped tests.  If you're interested in crash-coverage from skipped tests, I would suggest setting up a separate bot with one of those options.

No such resources are available, unfortunately. I also think additional systems could be used for better purpose, such as perfbots.


(In reply to comment #2)
> The Chromium port gets a large amount of crash-coverage from the huge fuzzing system which Inferno runs.  Each of those crashers is turned into a layout test and committed to svn.webkit.org for other ports to run.  I suspect that this "ClusterFuzz" provides infinitely more value than any ancient WONTFIX/Skipped, etc. layout test could.  IMO we should probably start culling old/broken layout tests which no one runs.  (We could also continue to ignore them in the noise, since I'm sure they're a tiny fraction of our 30k+ layout tests.)

Reading about ClusterFuzz on the Chromium blog[3], I understand (could be reading it incorrectly, though) that only exploitable crashes are analyzed and upstreamed. I appreciate the effort put into this system and process, but this in no way tests neither the WebKit1 layer of the GTK (or any other non-Chromium) port nor the WebKit2 layer.

[1] http://trac.webkit.org/browser/trunk/LayoutTests/fast/harness/sample-fail-mismatch-reftest.html
[2] http://trac.webkit.org/browser/trunk/LayoutTests/fast/harness/sample-fail-mismatch-reftest-expected-mismatch.html
[3] http://blog.chromium.org/2012/04/fuzzing-for-security.html

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.