[webkit-dev] Importing W3C tests to webkit

[Dirk Pranke]

On Wed, May 23, 2012 at 2:59 PM, Jacob Goldstein <jacobg at adobe.com> wrote:
> As a side note to this discussion, there is talk in the W3C community
> regarding their test approval process.  At the recent working group
> meetings in Germany the idea was floated to simply approve all tests that
> are currently waiting for review (and doing this going forward, e.g. no
> longer requiring approval upon submission).
>
> Apparently, not enough people are reviewing tests, and as a result, tests
> can linger for months (or longer) before ever being looked at.  Once
> browser vendors start implementing features, associated tests will be
> revisited for that area.
>
> This has by no means been decided, but something we should consider if it
> ultimately does come to fruition.  If W3C tests are no longer
> reviewed, this would mean importing tests without any knowledge of their
> accuracy - though that will still allow us to catch regressions.

Ick :(. I suppose if the test suites are mostly coming from
established browser vendors or other trusted sources (who have been
running them on their own browsers for a while) this isn't so bad, but
it would be nice to see the suites get vetted somehow prior to them
getting blessed by the w3c. Levels of approval, or something?

> Dirk, I've updated the process on the Wiki page with the feedback you
> provided.  I hope I captured it all - I included everything that appeared
> to have agreement between you and Ryosuke.  Feel free to modify it
> directly if I missed anything, or let me know and I can refine it further.

It looks much closer to what I'd like, thanks! I'll probably modify it
a bit and/or send you some suggestions offlist for some minor things.

The one major issue I still have that I wouldn't want to just
add/sneak into the wiki, is that I still don't see much of a
discussion for how we identify duplicate tests.

The mindset I would prefer would be that, for a given test suite, the
person or persons importing the test suite should have to start by
identifying which existing tests we have for similar functionality,
and produce lists of which existing tests look like they are
duplicates and should be remvoed, which tests we have that should be
submitted back to the w3c for future inclusion, and which tests are
clearly webkit-specific and should be kept but identified as clearly
webkit-specific.

For example, if we import a "flexbox" test suite from css3, we should
default to removing fast/flexbox (and possibly also
ietestcenter/css3/flexbox).

[ This is a purely hypothetical example (I know nothing about these
test suites). I wish to re-emphasize that we will have to look at
these things on a case-by-case basis. ]

I have no idea how much work we'd be asking the importer to undertake,
but that's kinda the point. If we don't ask the importer to de-dup
things, who is realistically going to de-dup them later?

What do others thinks, is this too much to ask? If it is, how do we
avoid incurring further "test debt" over time?

-- Dirk