[webkit-dev] Please don't leave entries for rebaseline in TestExpectation files

[Robert Hogan]

On Thursday, 21 March 2013, Ryosuke Niwa wrote:

> On Thu, Mar 21, 2013 at 1:31 AM, Robert Hogan <lists at roberthogan.net<javascript:_e({}, 'cvml', 'lists at roberthogan.net');>
> > wrote:
>
>> On Thursday, 21 March 2013, Ryosuke Niwa wrote:
>>
>>> I used to pull results from the bots where possible but creating
>>>> inconsistency between png/text results is not good.
>>>>
>>>
>>> It is unfortunate but it's much better than losing the complete test
>>> coverage.
>>>
>>
>> If that's the case then I'm happy to land whatever garden-o-matic pulls
>> in or I can sweep from the bots, even if it means that png results for Mac,
>> Qt, et al. go bad as a result.
>>
>> I guess we will always have ports whose bots do not run pixel tests so if
>> those ports are happy to live with the downsides of doing that then there
>> really is no obstacle to authors owning the job of updating the baselines
>> for all ports when they land a change.
>>
>> IMHO ports who don't run pixel tests would be better off deleting any png
>> results they have in the tree. Is there a reason Mac hasn't done that?
>> Don't you get lots of failures when you run pixel tests locally?
>>
>
> Yes, but I'd argue that it's better than losing the test coverage.
>
> By the way, we can easily address this problem by always generating pixel
> results for unexpectedly failing tests. Namely, we can force --pixel when
> we're retrying tests.
>
>
Perhaps NRWT could produce txt and png results for all tests marked with
REBASELINE or similar in TestExpectations. That would avoid the need to
turn the bots red on each platform for at least one build cycle.

Or is that something we can live with?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130321/f57a6234/attachment.html>