[webkit-dev] add NeedsRebaseline keyword to TestExpectations as a way to hande updating pixel tests?

[Dirk Pranke]

Makes sense to me ...

On Thu, Oct 4, 2012 at 2:46 PM, Ojan Vafai <ojan at chromium.org> wrote:
> TL;DR: We should add a NeedsRebaseline keyword to TestExpectations and add
> garden-o-matic tooling for it for the cases where someone commits a
> change/test that they know will need new results for different ports (e.g.
> any patch that changes the rendering of pixel tests).
>
> A common pattern that I see across ports is that someone will add something
> like the following in a patch that changes the results of a pixel test:
> // Needs rebaseline after r23456
> webkit.org/b/12345 path/to/test.html [ Failure ]
>
> This has a couple problems:
> -Often the correct expectation is something like [ Missing Failure
> ImageOnlyFailure ]. So, even though the test is listed, the bot turns red.
> -The tooling can't give you a list of all the tests that are expected to
> only need a rebaseline.
> -Related to the above, people often forget about these lines and don't do
> the rebaseline.
>
> We should add [ NeedsRebasline ], which is equivalent to [ Missing Failure
> ImageOnlyFailure ]. I'm thinking it should not include Timeout/Crash since
> those would need a solution other than a rebaseline (e.g. something is wrong
> with the test or patch).
>
> In garden-o-matic, we can make a tab specifically for tests that need
> rebaseline and give some indication whether the original patch that line was
> added in has run on all the relevant bots. This way the people keeping the
> tree green can also make sure that NeedsRebaseline lines don't get
> forgotten.
>
> If it continues to be a problem we could even setup and automated nag bot to
> email people who leave in NeedsRebaseline lines for more than a week.
>
> In the long-run, we should make it so you can grab the new results of the
> EWS bots and don't need to add lines to TestExpectations at all. In the
> short-term though, this is a way we can handle pixel tests without making
> the tree red all the time.
>
> As a side note, we should also get rid of Missing as a valid expectation. A
> test should either be NeedsRebaseline or have an expected result.
>
> Ojan
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo/webkit-dev
>