[webkit-dev] Increasing the number of cross-platform/port expected results

[Maciej Stachowiak]

On Feb 23, 2010, at 8:14 AM, Darin Adler wrote:

> On Feb 23, 2010, at 8:08 AM, Ariya Hidayat wrote:
>
>>> When the test is run, the browser loads both files, takes  
>>> snapshots, and does a pixel comparison. Thus font differences  
>>> between platforms become less of an issue.
>>
>> Isn't it our existing pixel test infrastructure? Or do you mean  
>> something else?
>
> In pixel tests, the expected file is a checked-in image. In  
> addition, we still have a render tree dump; it’s not just pixels.
>
> In reference tests, the expected file is HTML; different than the  
> test file, typically simple markup. Both files are rendered and the  
> resulting pixels are compared. And I presume there would be no  
> render tree dump.

The challenge with reftests is coming up with different markup that's  
supposed to render exactly the same, but for unrelated reasons. While  
it's possible, for example, to compare a table layout to absolute  
positioned boxes, for some things you get down to primitives that  
can't be tested independently. For example you can test that <b>,  
<strong> and font-weight: bold have the same results, but none of that  
tells you whether bold in fact works. You could have a regression  
which caused bold to be completely ignored and would not be able to  
catch it with a reftest. Even in cases where there are truly  
independent ways to get the same visual result, it can be tricky to  
design both.

That being said, it does do a good job of factoring out font  
differences.

Regards,
Maciej