[webkit-dev] Supporting w3c ref tests and changing our convention

[Alan Stearns]

On 11/4/11 7:20 PM, "Ojan Vafai" <ojan at chromium.org> wrote:

> On Fri, Nov 4, 2011 at 7:16 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> On Fri, Nov 4, 2011 at 7:07 PM, Ojan Vafai <ojan at chromium.org> wrote:
>>> On Fri, Nov 4, 2011 at 7:03 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>>> I am, but I'm particularly concerned about W3C tests. It'll be nice if we
>>>> had exactly one way of running / writing ref tests. I think we can easily
>>>> automate the process of generating manifest files.
>>>> 
>>>> The work flow will be as follows:
>>>> 1. Write new ref tests using link element
>>>> 2. Run some tool (maybe we can teach run-webkit-tests to do it
>>>> automatically)
>>>> 3. Upload patch with auto-regenerated manifest file
>>> It's mainly step 2 that I have a problem with, although I also don't like
>>> that the test is not self-contained.
>> 
>> Generating manifest file when we add a test is much more efficient than
>> generating it every time we run tests because we tend to do the latter much
>> more often than the former.
> 
> I think we're at an empass here. I don't see that further technical arguments
> will sway either of us. I do, however, expect that the vast majority of webkit
> developers would prefer to avoid a manifest file given the way the project has
> been structured up to now.
> 

What if we defer some of the W3C metadata work until tests were actually
submitted to the W3C?

1. Tests we pull from W3C can run from manifests, since they are provided.

2. Tests we develop ourselves just use a naming convention (refs are named
*-ref.html, and there's one ref per test even if that's duplicative)

3. When we choose to share a set of tests with the W3C, we do the extra work
of adding metadata to the tests and possibly refactoring to reduce the
number of -ref files. Once the W3C approves the tests we pull their copies
and delete ours.

Alan