[webkit-dev] Baselines ignoring metrics

[Dirk Pranke]

On Wed, Sep 19, 2012 at 2:00 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen <kde at carewolf.com> wrote:
>> On Wednesday 19 September 2012, Dirk Pranke wrote:
>>> After some limited amount of thought, I'm inclined to agree with
>>> Ryosuke and Ossy here. If we have tests that don't depend on checking
>>> the metrics, can they just be dumpAsText tests or reftests instead?
>>>
>> They could be. The tests I had in mind and made a conversion for are however
>> examples of tests from external test suites. We have many tests of this
>> nature, and converting them all the a webkit specific framework would be a lot
>> of work, no one would actually do, and would mean we could not share and
>> update the tests as easily with other members of W3C.
>>
>> This is exactly why I asked for what kind of guidelines we should make for
>> when we should use baselines without metrics. For instance we could decide new
>> tests cases made for WebKit bugs should alway use dumpAsText, and restrict
>> this feature to only test cases from an external source, or only older tests.
>>
>
> I see. My understanding is that the W3C is supposed to be moving in
> the direction of more reftests everywhere, so hopefully this becomes
> less of an issue. What to do with the legacy tests that may not get
> updated is a different issue, of course.
>
> My personal hope / theory / claim is that we should make the diff
> smarter about using the information we have to detect real issues,
> rather than throwing away information and then potentially missing
> issues.

Sorry, I realize this answer isn't as helpful as it could be.
Personally, I'm not a fan of adding new (external) test suites that
aren't text-only tests or reftests, and so I'm a bit negative here.
However, I think the value depends on the test suite in question and
what it's testing, so it's hard for me to give a blanket statement.

If you can point to specific tests you'd like to try this for I can
comment in more detail, but I'm not really okay with a blanket
approval for using --ignore-metrics whenever we feel like it yet.

-- Dirk