[webkit-dev] add NeedsRebaseline keyword to TestExpectations as a way to hande updating pixel tests?

[Ojan Vafai]

TL;DR: We should add a NeedsRebaseline keyword to TestExpectations and add
garden-o-matic tooling for it for the cases where someone commits a
change/test that they know will need new results for different ports (e.g.
any patch that changes the rendering of pixel tests).

A common pattern that I see across ports is that someone will add something
like the following in a patch that changes the results of a pixel test:
// Needs rebaseline after r23456
webkit.org/b/12345 path/to/test.html [ Failure ]

This has a couple problems:
-Often the correct expectation is something like [ Missing Failure
ImageOnlyFailure ]. So, even though the test is listed, the bot turns red.
-The tooling can't give you a list of all the tests that are expected to
only need a rebaseline.
-Related to the above, people often forget about these lines and don't do
the rebaseline.

We should add [ NeedsRebasline ], which is equivalent to [ Missing Failure
ImageOnlyFailure ]. I'm thinking it should not include Timeout/Crash since
those would need a solution other than a rebaseline (e.g. something is
wrong with the test or patch).

In garden-o-matic, we can make a tab specifically for tests that need
rebaseline and give some indication whether the original patch that line
was added in has run on all the relevant bots. This way the people keeping
the tree green can also make sure that NeedsRebaseline lines don't get
forgotten.

If it continues to be a problem we could even setup and automated nag bot
to email people who leave in NeedsRebaseline lines for more than a week.

In the long-run, we should make it so you can grab the new results of the
EWS bots and don't need to add lines to TestExpectations at all. In the
short-term though, this is a way we can handle pixel tests without making
the tree red all the time.

As a side note, we should also get rid of Missing as a valid expectation. A
test should either be NeedsRebaseline or have an expected result.

Ojan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20121004/2439c361/attachment.html>