[webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

[Jacob Goldstein]

I understand, all valid points.

Don't people currently fix tests that are failing and that they didn't author?  If so, isn't that similar as generating a new ref file in that you first have to understand what is being tested?  There would be the old pixel file to go from, so in many cases this may not be too hard.

On the other hand, having the test author create the ref file is definitely preferable.  What would we do in the case where a test author is no longer available?




From: Ryosuke Niwa <rniwa at webkit.org<mailto:rniwa at webkit.org>>
Date: Wed, 11 Apr 2012 16:00:44 -0700
To: Jacob Goldstein <jacobg at adobe.com<mailto:jacobg at adobe.com>>
Cc: Dirk Pranke <dpranke at chromium.org<mailto:dpranke at chromium.org>>, Robert Hogan <lists at roberthogan.net<mailto:lists at roberthogan.net>>, "webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>" <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>>
Subject: Re: [webkit-dev] importing test suites (was: CSS 2.1 Test Suite)

On Wed, Apr 11, 2012 at 3:52 PM, Jacob Goldstein <jacobg at adobe.com<mailto:jacobg at adobe.com>> wrote:
+1 on not introducing new pixel tests and allowing someone other than the
test author to create the -expected file.

We may also be able to streamline some of this process by implementing
some helper scripts.  Ultimately, someone will still have to review new
files manually, but scripts should be able to speed up the process.

-1 on that. As I said on other threads about this topic, determining whether a reference file adequately detect all bugs a test is intended to test is hard, and losing the test coverage at the cost of lowering maintenance cost is not necessary a good thing.

Also, adding a reference file would mean that either we're adding -ref.html / -noref.html files or modifying reftest.list. If doing the former, then we can't use this approach in any directory where we use reftest.list at the moment because we explicitly prohibit mixing naming convention and reftest.list.

Modifying reftest.list is essentially modifying the test suite, and it seems like there is a consensus that we don't want to do it.

- Ryosuke

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120411/81f907f4/attachment-0001.html>