[webkit-dev] Please don't leave entries for rebaseline in TestExpectation files

[Peter Kasting]

On Wed, Mar 20, 2013 at 11:46 PM, Robert Hogan <lists at roberthogan.net>wrote:

> On Wednesday, 20 March 2013, Ryosuke Niwa wrote:
>
>> Please don't add lines to TestExpectations saying that they just need
>> rebaselines and then leave.
>>
>
> OK. That means I will have to pull the new results from the bots, which is
> fine - but in the case of the Mac port (and any other bot that does not run
> pixel tests) the result will be that trunk will get fresh text results but
> retain stale png results.
>

I suspect by "and then leave" Ryosuke meant "and never come back".  It
seems reasonable to me to check in and then wait a sufficient amount of
time for the bots to cycle fully before using garden-o-matic to pull the
correct baselines.  This would mean we might have people leaving a
"[pkasting] Will rebaseline this test before Mar. 22, 2013" line on some
expectations for a day or two, just not forever.

 We've batted back and forth on this list for at least a year on the
> correct approach for landing and rebaselining. My approach is to land
> results for the platform that I build, suppress tests that require
> rebaselining on other platforms, and open a bug so sheriffs can
> add/rebaseline results as appropriate.
>

It's certainly nicer than not landing any expectations :).  But as the
current Chromium WebKit sheriff, I just spent a few hours rebaselining a
lot of these sorts of things in the Chromium expectations, some of which
had been around for months.  It's easy for these sorts of "needs
rebaseline" bugs to get lost in the shuffle, and in a few cases, I couldn't
determine if "needs rebaseline" was still correct due to further changes
that had happened since.  For these reasons, the original change author is
in the best position to ensure the right rebaselining happens quickly,
although I do realize that this is a nontrivial burden to place on change
authors.  I don't know if that means we should say "do this if you can, but
OK if not", or what.

My impression from recent discussion on this topic was that this was the
> way that worked best for everybody.I used to pull results from the bots
> where possible but creating inconsistency between png/text results is not
> good.
>

As long as the relevant bots have all cycled past the change in
question, your checkout contains all the relevant LayoutTest
subdirectories, and you've updated to ToT, I believe garden-o-matic can
correctly rebaseline any of the ports it supports, regardless of whether
you can build/run those ports locally.  Inconsistencies I've seen have been
a result of non-updated checkouts or non-cycled bots.

PK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130320/4ed2f12e/attachment.html>