[webkit-dev] generic test expectations?

[Dirk Pranke]

On Tue, Nov 13, 2012 at 12:16 PM, Tony Chang <tony at chromium.org> wrote:
> On Tue, Nov 13, 2012 at 12:04 PM, Darin Adler <darin at apple.com> wrote:
>>
>> On Nov 13, 2012, at 12:02 PM, Tony Chang <tony at chromium.org> wrote:
>>
>> > I don't think we should support port specific ref test results. That
>> > kind of misses the point of using a ref test in the first place. I mean, you
>> > may as well check in port specific pixel results which are easier to review
>> > for correctness.
>>
>> I don’t agree that pixel results are easier to review for correctness.
>
>
> Here is a ref test result from ietestcenter:
> http://trac.webkit.org/browser/trunk/LayoutTests/ietestcenter/css3/flexbox/flexbox-flex-002-expected.htm
>
> Looking at that HTML file, it's not immediately obvious that the result is
> correct.  If I had a png file, it would easy to see if there's red showing
> or not.
>

Actually reviewing the rendered pixels makes some things easier, but
it can be difficult to figure out which rendered pixels matter and
which don't. Even in the case of "if you see any red", it can be easy
to miss one red pixel on an edge (and of course, there's issues around
red-green colorblindness).

Plus, you have to consider both the cost of reviewing the initial
"correct output" and then the cost of reviewing changes to that
output. in theory, we'll get far fewer changes to the correct output
that need to be reviewed. For example, if we change how text is
rendered (cough *mountain lion* cough) we'll need a new PNG but no
change to the ref test. So, that's one less review you have to do.

That said, I don't think there's a blanket rule here; I suspect some
things are easier to review as pixel tests and some aren't.

-- Dirk