[webkit-dev] Importing W3C tests to webkit

[Ryosuke Niwa]

On Wed, May 23, 2012 at 1:25 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>
> Not so: if the tests we had had 100% coverage, then importing more
> tests would buy us nothing, but getting rid of the existing tests
> would be quite unfortunate.


We certainly don't have 100% test coverage.

Clearly adding tests incurs some costs and probably provides some benefit;
> the question is when does the cost exceed the benefit?
>

Doing anything incurs some cost. In fact, not adding a test itself incurs a
risk cost of potentially taking regressions in the future that could have
been caught by the test.

As I am not against importing more tests per se, I think this only
> makes sense to evaluate on a case-by-case basis.
>

Sure. We shouldn't be importing tests that are obviously duplicates of our
existing tests.

> The only sane argument I've heard so far to gate pixel tests is that the
> > correctness of such tests need to be manually inspected, which requires a
> > lot of manual labor and is very error prone.
>
> I'm assuming the above includes the ongoing maintenance cost of
> keeping pixel tests up to date, as well as the cost at the initial
> checkin.


I'm not concerned of those. Once the correct expected result is checked in,
it's pretty easy to rebaseline tests per rendering engine changes assuming
people who are rebaselining tests know what they're doing.

There is also the fact that the more tests we have, the more tests we
> have to run, and increasing cycle time by itself is a cost to developer
> productivity.


Sure, but I don't think that's a valid argument for not adding tests
especially since there is no way for us to mechanically test whether two
tests test the same set of features or not (this is an intractable problem
even in its limited form and an undecidable one in its most general form).

Also, using ref test or pixel test, etc... doesn't change the cycle time
significantly so I don't understand what your argument is. Or are you
suggesting that non-ref tests are somehow more redundant than ref tests?
(please give us why).

Of course, it's also potentially the case that
> we have to update tests from time to time, although this doesn't
> happen often.
>

Because the current process is broken :) In the ideal world, we would be
updating our copy of W3C tests every so often (e.g. every month or so).

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120523/4bca312b/attachment.html>