[webkit-dev] Pixel test experiment

[Dirk Schulze]

> The problem I worry about is that on future Mac OS X releases, rendering of shapes may change in some tiny way that is not visible but enough to cause failures at tolerance 0. In the past, such false positives arose from time to time, which is one reason we added pixel test tolerance in the first place. I don't think running pixel tests on just one build slave will help us understand that risk.
> 
> Why not start with some low but non-zero tolerance (0.1?) and see if we can at least make that work consistently, before we try the bolder step of tolerance 0?
> 
> Also, and as a side note, we probably need to add more build slaves to run pixel tests at all, since just running the test suite without pixel tests is already slow enough that the testers are often significantly behind the builders.
> 
> Regards,
> Maciej
Running pixel test with a tolerance of 0.1 is still better than don't run pixel tests at all. So if we get a consensus with a small tolerance, I'm fine.
And yes, we might get problems with a new MacOS release. We have a lot of differences (<0.1%) between 10.5 and 10.6 right now.
But I don't see a problem with it as long as someone manages the results. Niko and I are doing it for SVG on MacOSX 10.6 and also continue it on 10.5 for a while.

Dirk