[webkit-dev] Pixel test experiment

[Nikolas Zimmermann]

Am 08.10.2010 um 19:53 schrieb Maciej Stachowiak:

>
> On Oct 8, 2010, at 12:46 AM, Nikolas Zimmermann wrote:
>
>>
>> Am 08.10.2010 um 00:44 schrieb Maciej Stachowiak:
>>
>>>
>>> On Oct 7, 2010, at 6:34 AM, Nikolas Zimmermann wrote:
>>>
>>>> Good evening webkit folks,
>>>>
>>>> I've finished landing svg/ pixel test baselines, which pass with  
>>>> --tolerance 0 on my 10.5 & 10.6 machines.
>>>> As the pixel testing is very important for the SVG tests, I'd  
>>>> like to run them on the bots, experimentally, so we can catch  
>>>> regressions easily.
>>>>
>>>> Maybe someone with direct access to the leopard & snow leopard  
>>>> bots, could just run "run-webkit-tests --tolerance 0 -p svg" and  
>>>> mail me the results?
>>>> If it passes, we could maybe run the pixel tests for the svg/  
>>>> subdirectory on these bots?
>>>
>>> Running pixel tests would be great, but can we really expect the  
>>> results to be stable cross-platform with tolerance 0? Perhaps we  
>>> should start with a higher tolerance level.
>>
>> Sure, we could do that. But I'd really like to get a feeling, for  
>> what's problematic first. If we see 95% of the SVG tests pass with  
>> --tolerance 0, and only a few need higher tolerances
>> (64bit vs. 32bit aa differences, etc.), I could come up with a per- 
>> file pixel test tolerance extension to DRT, if it's needed.
>>
>> How about starting with just one build slave (say. Mac Leopard)  
>> that runs the pixel tests for SVG, with --tolerance 0 for a while.  
>> I'd be happy to identify the problems, and see
>> if we can make it work, somehow :-)
>
> The problem I worry about is that on future Mac OS X releases,  
> rendering of shapes may change in some tiny way that is not visible  
> but enough to cause failures at tolerance 0. In the past, such false  
> positives arose from time to time, which is one reason we added  
> pixel test tolerance in the first place. I don't think running pixel  
> tests on just one build slave will help us understand that risk.

I think we'd just update the baseline to the newer OS X release, then,  
like it has been done for the tiger -> leopard, leopard -> snow  
leopard switch?
platform/mac/ should always contain the newest release baseline, when  
therere are differences on leopard, the results go into platform/mac- 
leopard/

> Why not start with some low but non-zero tolerance (0.1?) and see if  
> we can at least make that work consistently, before we try the  
> bolder step of tolerance 0?
> Also, and as a side note, we probably need to add more build slaves  
> to run pixel tests at all, since just running the test suite without  
> pixel tests is already slow enough that the testers are often  
> significantly behind the builders.

Well, I thought about just running the pixel tests for the svg/  
subdirectory as a seperate step, hence my request for tolerance 0, as  
the baseline passes without problems at least on my & Dirks machine  
already.
I wouldnt' want to argue running 20.000+ pixel tests with tolerance 0  
as first step :-) But the 1000 SVG tests, might be fine, with  
tolerance 0?

Even tolerance 0.1 as default for SVG would be fine with me, as long  
as we can get the bots to run the SVG pixel tests :-)

Cheers,
Niko