[webkit-dev] pixel tests and --tolerance (was Re: Pixel test experiment)

[Simon Fraser]

I think the best solution to this pixel matching problem is ref tests.

How practical would it be to use ref tests for SVG?

Simon

On Oct 8, 2010, at 12:43 PM, Dirk Pranke wrote:

> Jeremy is correct; the Chromium port has seen real regressions that
> virtually no concept of a fuzzy match that I can imagine would've
> caught.
> new-run-webkit-tests doesn't currently support the tolerance concept
> at al, and I am inclined to argue that it shouldn't.
> 
> However, I frequently am wrong about things, so it's quite possible
> that there are good arguments for supporting it that I'm not aware of.
> I'm not particularly interested in working on a tool that doesn't do
> what the group wants it to do, and I would like all of the other
> WebKit ports to be running pixel tests by default (and
> new-run-webkit-tests ;) ) since I think it catches bugs.
> 
> As far as I know, the general sentiment on the list has been that we
> should be running pixel tests by default, and the reason that we
> aren't is largely due to the work involved in getting them back up to
> date and keeping them up to date. I'm sure that fuzzy matching reduces
> the work load, especially for the sort of mismatches caused by
> differences in the text antialiasing.
> 
> In addition, I have heard concerns that we'd like to keep fuzzy
> matching because people might potentially get different results on
> machines with different hardware configurations, but I don't know that
> we have any confirmed cases of that (except for arguably the case of
> different code paths for gpu-accelerated rendering vs. unaccelerated
> rendering).
> 
> If we made it easier to maintain the baselines (improved tooling like
> the chromium's rebaselining tool, add reftest support, etc.) are there
> still compelling reasons for supporting --tolerance -based testing as
> opposed to exact matching?
> 
> -- Dirk
> 
> On Fri, Oct 8, 2010 at 11:14 AM, Jeremy Orlow <jorlow at chromium.org> wrote:
>> I'm not an expert on Pixel tests, but my understanding is that in Chromium
>> (where we've always run with tolerance 0) we've seen real regressions that
>> would have slipped by with something like tolerance 0.1.  When you have
>> 0 tolerance, it is more maintenance work, but if we can avoid regressions,
>> it seems worth it.
>> J
>> 
>> On Fri, Oct 8, 2010 at 10:58 AM, Nikolas Zimmermann
>> <zimmermann at physik.rwth-aachen.de> wrote:
>>> 
>>> Am 08.10.2010 um 19:53 schrieb Maciej Stachowiak:
>>> 
>>>> 
>>>> On Oct 8, 2010, at 12:46 AM, Nikolas Zimmermann wrote:
>>>> 
>>>>> 
>>>>> Am 08.10.2010 um 00:44 schrieb Maciej Stachowiak:
>>>>> 
>>>>>> 
>>>>>> On Oct 7, 2010, at 6:34 AM, Nikolas Zimmermann wrote:
>>>>>> 
>>>>>>> Good evening webkit folks,
>>>>>>> 
>>>>>>> I've finished landing svg/ pixel test baselines, which pass with
>>>>>>> --tolerance 0 on my 10.5 & 10.6 machines.
>>>>>>> As the pixel testing is very important for the SVG tests, I'd like to
>>>>>>> run them on the bots, experimentally, so we can catch regressions easily.
>>>>>>> 
>>>>>>> Maybe someone with direct access to the leopard & snow leopard bots,
>>>>>>> could just run "run-webkit-tests --tolerance 0 -p svg" and mail me the
>>>>>>> results?
>>>>>>> If it passes, we could maybe run the pixel tests for the svg/
>>>>>>> subdirectory on these bots?
>>>>>> 
>>>>>> Running pixel tests would be great, but can we really expect the
>>>>>> results to be stable cross-platform with tolerance 0? Perhaps we should
>>>>>> start with a higher tolerance level.
>>>>> 
>>>>> Sure, we could do that. But I'd really like to get a feeling, for what's
>>>>> problematic first. If we see 95% of the SVG tests pass with --tolerance 0,
>>>>> and only a few need higher tolerances
>>>>> (64bit vs. 32bit aa differences, etc.), I could come up with a per-file
>>>>> pixel test tolerance extension to DRT, if it's needed.
>>>>> 
>>>>> How about starting with just one build slave (say. Mac Leopard) that
>>>>> runs the pixel tests for SVG, with --tolerance 0 for a while. I'd be happy
>>>>> to identify the problems, and see
>>>>> if we can make it work, somehow :-)
>>>> 
>>>> The problem I worry about is that on future Mac OS X releases, rendering
>>>> of shapes may change in some tiny way that is not visible but enough to
>>>> cause failures at tolerance 0. In the past, such false positives arose from
>>>> time to time, which is one reason we added pixel test tolerance in the first
>>>> place. I don't think running pixel tests on just one build slave will help
>>>> us understand that risk.
>>> 
>>> I think we'd just update the baseline to the newer OS X release, then,
>>> like it has been done for the tiger -> leopard, leopard -> snow leopard
>>> switch?
>>> platform/mac/ should always contain the newest release baseline, when
>>> therere are differences on leopard, the results go into
>>> platform/mac-leopard/
>>> 
>>>> Why not start with some low but non-zero tolerance (0.1?) and see if we
>>>> can at least make that work consistently, before we try the bolder step of
>>>> tolerance 0?
>>>> Also, and as a side note, we probably need to add more build slaves to
>>>> run pixel tests at all, since just running the test suite without pixel
>>>> tests is already slow enough that the testers are often significantly behind
>>>> the builders.
>>> 
>>> Well, I thought about just running the pixel tests for the svg/
>>> subdirectory as a seperate step, hence my request for tolerance 0, as the
>>> baseline passes without problems at least on my & Dirks machine already.
>>> I wouldnt' want to argue running 20.000+ pixel tests with tolerance 0 as
>>> first step :-) But the 1000 SVG tests, might be fine, with tolerance 0?
>>> 
>>> Even tolerance 0.1 as default for SVG would be fine with me, as long as we
>>> can get the bots to run the SVG pixel tests :-)
>>> 
>>> Cheers,
>>> Niko
>>> 
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> 
>> 
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>> 
>> 
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev