[webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

[Dirk Pranke]

On Fri, Feb 8, 2013 at 11:12 AM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Fri, Feb 8, 2013 at 5:35 AM, <noam.rosenthal at nokia.com> wrote:
>>
>> we don't currently have a solution in webkit's test infrastructure for
>> testing animations "on the fly", or in general for testing multiple image
>> results in the same test.
>> (If this was discussed before and I'm unaware of that discussion, please
>> stop me here...)
>>
>> This is because ImageDiff works on the single-test level, and you can't
>> explicitly make image comparisons in the same test at different points in
>> time.
>> This has before caused us several regressions that were not caught by
>> layout tests.
>>
>> I'd like to propose a solution, and would welcome some feedback on whether
>> it's a good one...
>> The idea is that you would be able to programatically retrieve the current
>> snapshot into a canvas ImageData, and then compare the pixel results with
>> JavaScript in the LayoutTest. Something like:
>
>
> I had similar thoughts but my counter "proposal" is to let DRT/WTR generate
> multiple actual results either in the form of multiple layers in PNG or
> multiple PNG images. The advantage of this latter approach is that we can
> have multiple reference files as well. e.g. if a green box is to be moved
> from (0,0) to (50,0) and to (100, 0), we could create three reference files
> that correspond to each state.
>
> But perhaps there is a good reason you didn't choose this approach. Could
> you elaborate on the reason you picked this particular API?
>

I also appreciate the problem -- and would be interesting in solving
it -- but I'm a bit concerned that either proposal is at least
somewhat nondeterministic (or flaky), and I'm not sure how you'd be
able to control which frames of the animation you were getting well
enough. But I'm not much of expert here so I'm not sure how often it
matters.

-- Dirk