[webkit-dev] Importing W3C tests to webkit

[Maciej Stachowiak]

On May 23, 2012, at 2:16 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>>> The only sane argument I've heard so far to gate pixel tests is that the
>>>> correctness of such tests need to be manually inspected, which requires
>>>> a
>>>> lot of manual labor and is very error prone.
>>> 
>>> I'm assuming the above includes the ongoing maintenance cost of
>>> keeping pixel tests up to date, as well as the cost at the initial
>>> checkin.
>> 
>> I'm not concerned of those. Once the correct expected result is checked in,
>> it's pretty easy to rebaseline tests per rendering engine changes assuming
>> people who are rebaselining tests know what they're doing.
> 
> You should be concerned; keeping pixel tests up-to-date is clearly a
> non-zero cost that only the chromium port thus far has been willing to
> bear, and I suspect that the cost of updating baselines is
> substantially higher than the cost of the initial review over time
> (since it's a recurring cost).

Are you concerned just about the actual pixel results or also about keeping render tree dumps up to date? We can address the pixel result issue by introducing a new test that dumps its render tree but does not do pixel testing.

I think there is a high value to importing standards test suites wholesale, even if they overlap with our existing coverage. Picking and choosing subsets makes things  more complicated. If there are significant externalities to adding particular kinds of tests, I would prefer we mitigate those externalities rather than run fewer tests.

Regards,
Maciej