[webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

[Dirk Pranke]

On Sat, Feb 9, 2013 at 1:57 AM, Benjamin Poulain <benjamin at webkit.org> wrote:
> On Sat, Feb 9, 2013 at 1:24 AM, <noam.rosenthal at nokia.com> wrote:
>>
>> I did not say they cannot be tested with the two methods suggested :) It's
>> more  about a preference to move some of those decisions from the test
>> infrastructure to the test itself, but potentially those bugs could be
>> tested in either way.
>> Examples for bugs I've encountered/reviewed and can use better in-motion
>> testing (note that those are specific to Qt/EFL, but I'm sure there are tons
>> of bugs like this that come up for Apple/Google as well)
>> http://trac.webkit.org/changeset/140825
>> http://trac.webkit.org/changeset/142112
>> http://trac.webkit.org/changeset/134953
>> https://bugs.webkit.org/show_bug.cgi?id=109179
>>
>> Controlling the clock and programmatically sampling the end result would
>> definitely make those more testable, but of course any progress in this area
>> would be beneficial and my preference to a canvas-based API is more of an
>> opinion.
>
>
> To explain my concerns:
> Sometime, I look at a failing test, and think: "what the f**k is this
> supposed to test?". Then I have to dig a ton of logs, and finally read the
> change to understand a the single JS statement in the whole script that make
> the test useful.
>
> This is the situation I am afraid with a solution where pixels would be
> evaluated from JavaScript. You can test, but you lack visibility why
> something is correct or incorrect. Text tests, ref-tests and pixel tests
> have a great readability, you can quickly evaluate their correctness. This
> is important in my opinion, I don't think we want more opaque output like
> the RenderTree dump.
>
> I am not against your plan if the readability of the tests can be good. I'd
> be happy to review patches toward testing dynamic changes in webpages.
>

A counter-argument, of course, that there are a lot of pixel tests
that would be a lot better as text-only tests that were verifying
certain aspects of how the page rendered programmatically.

Perhaps -- and this is a tangent to this thread -- we could also
investigate ways of writing tests that did render pngs but told NRWT
to ignore them (e.g., dumpAsTextAndIgnoredImage() or something), or
conveyed which parts of the image were important ...

-- Dirk