[webkit-dev] Importing W3C tests to webkit

[Jacob Goldstein]

Dirk, my apologies, I was on travel the week you replied and missed your
message.  I found it and will review / update now.



On 5/23/12 1:25 PM, "Dirk Pranke" <dpranke at chromium.org> wrote:

>On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> As I have said in the past, we should just import all tests, and treat
>> non-text, non-ref tests as pixel tests. If we wanted to reduce the
>>number of
>> pixel tests we import, then we should submit those patches to W3C
>>instead of
>> directly submitting them to WebKit.
>>
>> In general, I don't buy the argument that adding more pixel tests incurs
>> more cost than the benefit we get from importing the tests. If that
>>were the
>> case, we can just get rid of the existing pixel tests we have.
>>
>
>Not so: if the tests we had had 100% coverage, then importing more
>tests would buy us nothing, but getting rid of the existing tests
>would be quite unfortunate. Clearly adding tests incurs some costs and
>probably provides some benefit; the question is when does the cost
>exceed the benefit?
>
>As I am not against importing more tests per se, I think this only
>makes sense to evaluate on a case-by-case basis.
>
>> The only sane argument I've heard so far to gate pixel tests is that the
>> correctness of such tests need to be manually inspected, which requires
>>a
>> lot of manual labor and is very error prone.
>>
>
>I'm assuming the above includes the ongoing maintenance cost of
>keeping pixel tests up to date, as well as the cost at the initial
>checkin.
>
>There is also the fact that the more tests we have, the more tests we
>have to run, and increasing cycle time by itself is a cost to
>developer productivity. Of course, it's also potentially the case that
>we have to update tests from time to time, although this doesn't
>happen often.
>
>Jacob, I gave you a bunch of feedback not long after you published the
>initial writeup, but it doesn't look like any of that has been
>incorporated?
>
>-- Dirk