[webkit-dev] Please don't leave entries for rebaseline in TestExpectation files

[Ryosuke Niwa]

On Thu, Mar 21, 2013 at 10:54 AM, Žan Doberšek <zandobersek at gmail.com>wrote:

> On Thu, Mar 21, 2013 at 5:18 PM, Robert Hogan <lists at roberthogan.net>wrote:
>
>> On Thursday, 21 March 2013, Ryosuke Niwa wrote:
>>
>>> On Thu, Mar 21, 2013 at 1:31 AM, Robert Hogan <lists at roberthogan.net>wrote:
>>>
>>>> On Thursday, 21 March 2013, Ryosuke Niwa wrote:
>>>>
>>>>>  I used to pull results from the bots where possible but creating
>>>>>> inconsistency between png/text results is not good.
>>>>>>
>>>>>
>>>>> It is unfortunate but it's much better than losing the complete test
>>>>> coverage.
>>>>>
>>>>
>>>> If that's the case then I'm happy to land whatever garden-o-matic pulls
>>>> in or I can sweep from the bots, even if it means that png results for Mac,
>>>> Qt, et al. go bad as a result.
>>>>
>>>> I guess we will always have ports whose bots do not run pixel tests so
>>>> if those ports are happy to live with the downsides of doing that then
>>>> there really is no obstacle to authors owning the job of updating the
>>>> baselines for all ports when they land a change.
>>>>
>>>> IMHO ports who don't run pixel tests would be better off deleting any
>>>> png results they have in the tree. Is there a reason Mac hasn't done that?
>>>> Don't you get lots of failures when you run pixel tests locally?
>>>>
>>>
>>> Yes, but I'd argue that it's better than losing the test coverage.
>>>
>>> By the way, we can easily address this problem by always generating
>>> pixel results for unexpectedly failing tests. Namely, we can force --pixel
>>> when we're retrying tests.
>>>
>>>
>> Perhaps NRWT could produce txt and png results for all tests marked with
>> REBASELINE or similar in TestExpectations. That would avoid the need to
>> turn the bots red on each platform for at least one build cycle.
>>
>
> I like this specific proposal. There's already a similar expectation
> planned, 'NeedsRebaseline'.
> https://bugs.webkit.org/show_bug.cgi?id=100415
>

How do we know that new results is correct prior to running tests on each
platform/port?  There are cases where we regress tests on some ports while
needing to rebaseline on other ports but all of that is unknown until we
actually run tests on the bots.

- R. Niwa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130321/65c2cde2/attachment-0001.html>