[webkit-dev] Please don't leave entries for rebaseline in TestExpectation files

[Žan Doberšek]

On Thu, Mar 21, 2013 at 5:18 PM, Robert Hogan <lists at roberthogan.net> wrote:

>
>
> On Thursday, 21 March 2013, Ryosuke Niwa wrote:
>
>> On Thu, Mar 21, 2013 at 1:31 AM, Robert Hogan <lists at roberthogan.net>wrote:
>>
>>> On Thursday, 21 March 2013, Ryosuke Niwa wrote:
>>>
>>>> I used to pull results from the bots where possible but creating
>>>>> inconsistency between png/text results is not good.
>>>>>
>>>>
>>>> It is unfortunate but it's much better than losing the complete test
>>>> coverage.
>>>>
>>>
>>> If that's the case then I'm happy to land whatever garden-o-matic pulls
>>> in or I can sweep from the bots, even if it means that png results for Mac,
>>> Qt, et al. go bad as a result.
>>>
>>> I guess we will always have ports whose bots do not run pixel tests so
>>> if those ports are happy to live with the downsides of doing that then
>>> there really is no obstacle to authors owning the job of updating the
>>> baselines for all ports when they land a change.
>>>
>>> IMHO ports who don't run pixel tests would be better off deleting any
>>> png results they have in the tree. Is there a reason Mac hasn't done that?
>>> Don't you get lots of failures when you run pixel tests locally?
>>>
>>
>> Yes, but I'd argue that it's better than losing the test coverage.
>>
>> By the way, we can easily address this problem by always generating pixel
>> results for unexpectedly failing tests. Namely, we can force --pixel when
>> we're retrying tests.
>>
>>
> Perhaps NRWT could produce txt and png results for all tests marked with
> REBASELINE or similar in TestExpectations. That would avoid the need to
> turn the bots red on each platform for at least one build cycle.
>

I like this specific proposal. There's already a similar expectation
planned, 'NeedsRebaseline'.
https://bugs.webkit.org/show_bug.cgi?id=100415


>
> Or is that something we can live with?
>
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130321/f6c056b6/attachment.html>