[webkit-dev] Importing W3C tests to webkit

[Jacob Goldstein]

On 5/23/12 2:30 PM, "Maciej Stachowiak" <mjs at apple.com> wrote:

>
>On May 23, 2012, at 2:16 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>
>> On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>>>> The only sane argument I've heard so far to gate pixel tests is that
>>>>>the
>>>>> correctness of such tests need to be manually inspected, which
>>>>>requires
>>>>> a
>>>>> lot of manual labor and is very error prone.
>>>> 
>>>> I'm assuming the above includes the ongoing maintenance cost of
>>>> keeping pixel tests up to date, as well as the cost at the initial
>>>> checkin.
>>> 
>>> I'm not concerned of those. Once the correct expected result is
>>>checked in,
>>> it's pretty easy to rebaseline tests per rendering engine changes
>>>assuming
>>> people who are rebaselining tests know what they're doing.
>> 
>> You should be concerned; keeping pixel tests up-to-date is clearly a
>> non-zero cost that only the chromium port thus far has been willing to
>> bear, and I suspect that the cost of updating baselines is
>> substantially higher than the cost of the initial review over time
>> (since it's a recurring cost).
>
>Are you concerned just about the actual pixel results or also about
>keeping render tree dumps up to date? We can address the pixel result
>issue by introducing a new test that dumps its render tree but does not
>do pixel testing.
>
>I think there is a high value to importing standards test suites
>wholesale, even if they overlap with our existing coverage. Picking and
>choosing subsets makes things  more complicated. If there are significant
>externalities to adding particular kinds of tests, I would prefer we
>mitigate those externalities rather than run fewer tests.


As a side note to this discussion, there is talk in the W3C community
regarding their test approval process.  At the recent working group
meetings in Germany the idea was floated to simply approve all tests that
are currently waiting for review (and doing this going forward, e.g. no
longer requiring approval upon submission).

Apparently, not enough people are reviewing tests, and as a result, tests
can linger for months (or longer) before ever being looked at.  Once
browser vendors start implementing features, associated tests will be
revisited for that area.

This has by no means been decided, but something we should consider if it
ultimately does come to fruition.  If W3C tests are no longer
reviewed, this would mean importing tests without any knowledge of their
accuracy - though that will still allow us to catch regressions.



Dirk, I've updated the process on the Wiki page with the feedback you
provided.  I hope I captured it all - I included everything that appeared
to have agreement between you and Ryosuke.  Feel free to modify it
directly if I missed anything, or let me know and I can refine it further.


Jacob


>
>Regards,
>Maciej
>
>_______________________________________________
>webkit-dev mailing list
>webkit-dev at lists.webkit.org
>http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev