[webkit-dev] Supporting w3c ref tests and changing our convention

[Ryosuke Niwa]

On Fri, Nov 4, 2011 at 4:01 PM, Ojan Vafai <ojan at chromium.org> wrote:

> I don't see any need for manifest files.


W3C's build step auto-generates them.

On Fri, Nov 4, 2011 at 2:00 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>> On Fri, Nov 4, 2011 at 1:47 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>>
>>> It's unclear how much of a perf impact there would be but that's
>>> easy enough to determine - I would expect it to be minimal compared to
>>> the time of actually rendering a page.
>>>
>>
>> Since I expect w3c to end up having hundreds of thousands of tests, I see
>> any performance implication to be a serious threat.
>>
>
> There is no inherent performance problem that I can see. We just need to
> structure things in a way that avoids performance problems.
>

But we don't need to structure them. They're organized in W3C's repo, and
we're just going to import them in some directory; e.g. LayoutTests/w3c/.
If we're adding new tests that we intend to contribute back to W3C, then
those tests should live outside of this directory. The preferred approach,
however, is to add tests to W3C repo first, then import them back to WebKit.

This only requires that we ensure that all reference files are either
> themselves tests or have "ref" in the name and/or path. Even if the W3C
> and/or Mozilla test suites don't end up enforcing this requirement we can
> check in a script to Tools/Scripts that restructures the tests
> appropriately before we commit them to our tree.
>

That, to me, is a problem. We try not to modify external tests or
restructure them when we import them. Renaming files will also make it hard
to figure out the correspondence between tests in WebKit's repo and W3C's
repo. It's a significant cognitive stress if one is to contribute tests
back to W3C's test suite.

The performance benefits of the manifest files could be addressed by not
> walking the tree before running the tests. We only do that now because that
> was the expedient way to write the code. If walking the tree turns out to
> be a performance problem down the road, we can run the tests as we walk the
> tree. If that's still too slow, we can generate a transient manifest file
> that goes in your output directory.
>

Generating a transient manifest file may make sense for WebKit tests but
not for W3C tests although I don't see why generating a transient manifest
file should improve the performance.

Also, I don't see why we should be generating manifest files on demand when
W3C test suite's build step automatically does that for us. We can build &
check tests into the repo once and we'll have manifest files. Since we
shouldn't be modifying those tests anyway, I don't see why we need to
generate manifests on demand.

I don't see what's special about reftests that we'd run them on a separate
> bot. We might decide to shard test running across different bots in some
> way, but sharding by test type seems unhelpful.
>

I'm not talking about ref-tests. I'm talking about tests imported from the
W3C test suite.

- Ryosuke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111104/7cbdd426/attachment.html>