[webkit-dev] Baselines ignoring metrics

[Dirk Pranke]

On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen <kde at carewolf.com> wrote:
> On Wednesday 19 September 2012, Dirk Pranke wrote:
>> After some limited amount of thought, I'm inclined to agree with
>> Ryosuke and Ossy here. If we have tests that don't depend on checking
>> the metrics, can they just be dumpAsText tests or reftests instead?
>>
> They could be. The tests I had in mind and made a conversion for are however
> examples of tests from external test suites. We have many tests of this
> nature, and converting them all the a webkit specific framework would be a lot
> of work, no one would actually do, and would mean we could not share and
> update the tests as easily with other members of W3C.
>
> This is exactly why I asked for what kind of guidelines we should make for
> when we should use baselines without metrics. For instance we could decide new
> tests cases made for WebKit bugs should alway use dumpAsText, and restrict
> this feature to only test cases from an external source, or only older tests.
>

I see. My understanding is that the W3C is supposed to be moving in
the direction of more reftests everywhere, so hopefully this becomes
less of an issue. What to do with the legacy tests that may not get
updated is a different issue, of course.

My personal hope / theory / claim is that we should make the diff
smarter about using the information we have to detect real issues,
rather than throwing away information and then potentially missing
issues.

-- Dirk