[webkit-dev] Supporting w3c ref tests and changing our convention

[Ojan Vafai]

I don't see any need for manifest files. Needing to maintain information
about a test somewhere other than in the test itself or in the expected
result file is a significant maintenance burden. We should avoid it if we
can.

On Fri, Nov 4, 2011 at 2:00 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:

> On Fri, Nov 4, 2011 at 1:47 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> It's unclear how much of a perf impact there would be but that's
>> easy enough to determine - I would expect it to be minimal compared to
>> the time of actually rendering a page.
>>
>
> Since I expect w3c to end up having hundreds of thousands of tests, I see
> any performance implication to be a serious threat.
>

There is no inherent performance problem that I can see. We just need to
structure things in a way that avoids performance problems. This only
requires that we ensure that all reference files are either themselves
tests or have "ref" in the name and/or path. Even if the W3C and/or Mozilla
test suites don't end up enforcing this requirement we can check in a
script to Tools/Scripts that restructures the tests appropriately before we
commit them to our tree.

The performance benefits of the manifest files could be addressed by not
walking the tree before running the tests. We only do that now because that
was the expedient way to write the code. If walking the tree turns out to
be a performance problem down the road, we can run the tests as we walk the
tree. If that's still too slow, we can generate a transient manifest file
that goes in your output directory.

Separately, if we are throwing around numbers in the range of >100K
>
>  for tests to run, we should consider when we actually want to run them
>> - i.e., what will the cycle time be if we run them on every change,
>> etc.? But that can be dealt with when we get there.
>>
>
> We need separate bots in the long term for sure.
>

I don't see what's special about reftests that we'd run them on a separate
bot. We might decide to shard test running across different bots in some
way, but sharding by test type seems unhelpful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20111104/fcaa56af/attachment.html>