[webkit-dev] Importing W3C tests to webkit

[Dirk Pranke]

On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> As I have said in the past, we should just import all tests, and treat
> non-text, non-ref tests as pixel tests. If we wanted to reduce the number of
> pixel tests we import, then we should submit those patches to W3C instead of
> directly submitting them to WebKit.
>
> In general, I don't buy the argument that adding more pixel tests incurs
> more cost than the benefit we get from importing the tests. If that were the
> case, we can just get rid of the existing pixel tests we have.
>

Not so: if the tests we had had 100% coverage, then importing more
tests would buy us nothing, but getting rid of the existing tests
would be quite unfortunate. Clearly adding tests incurs some costs and
probably provides some benefit; the question is when does the cost
exceed the benefit?

As I am not against importing more tests per se, I think this only
makes sense to evaluate on a case-by-case basis.

> The only sane argument I've heard so far to gate pixel tests is that the
> correctness of such tests need to be manually inspected, which requires a
> lot of manual labor and is very error prone.
>

I'm assuming the above includes the ongoing maintenance cost of
keeping pixel tests up to date, as well as the cost at the initial
checkin.

There is also the fact that the more tests we have, the more tests we
have to run, and increasing cycle time by itself is a cost to
developer productivity. Of course, it's also potentially the case that
we have to update tests from time to time, although this doesn't
happen often.

Jacob, I gave you a bunch of feedback not long after you published the
initial writeup, but it doesn't look like any of that has been
incorporated?

-- Dirk