[webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

[noam.rosenthal at nokia.com]

Since people seem to agree that this is a good problem to solve, I've created a bug for it: https://bugs.webkit.org/show_bug.cgi?id=109356
I can't promise to fix it myself right this moment as I'm spread a bit too thin, but if someone wants to help or pick it up before I get to it please feel free :)
Noam

From: ext Benjamin Poulain <benjamin at webkit.org<mailto:benjamin at webkit.org>>
Date: Saturday, February 9, 2013 10:57 AM
To: Noam Rosenthal <noam.rosenthal at nokia.com<mailto:noam.rosenthal at nokia.com>>
Cc: "rniwa at webkit.org<mailto:rniwa at webkit.org>" <rniwa at webkit.org<mailto:rniwa at webkit.org>>, "webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>" <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>>
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

On Sat, Feb 9, 2013 at 1:24 AM, <noam.rosenthal at nokia.com<mailto:noam.rosenthal at nokia.com>> wrote:
I did not say they cannot be tested with the two methods suggested :) It's more  about a preference to move some of those decisions from the test infrastructure to the test itself, but potentially those bugs could be tested in either way.
Examples for bugs I've encountered/reviewed and can use better in-motion testing (note that those are specific to Qt/EFL, but I'm sure there are tons of bugs like this that come up for Apple/Google as well)
http://trac.webkit.org/changeset/140825
http://trac.webkit.org/changeset/142112
http://trac.webkit.org/changeset/134953
https://bugs.webkit.org/show_bug.cgi?id=109179

Controlling the clock and programmatically sampling the end result would definitely make those more testable, but of course any progress in this area would be beneficial and my preference to a canvas-based API is more of an opinion.

To explain my concerns:
Sometime, I look at a failing test, and think: "what the f**k is this supposed to test?". Then I have to dig a ton of logs, and finally read the change to understand a the single JS statement in the whole script that make the test useful.

This is the situation I am afraid with a solution where pixels would be evaluated from JavaScript. You can test, but you lack visibility why something is correct or incorrect. Text tests, ref-tests and pixel tests have a great readability, you can quickly evaluate their correctness. This is important in my opinion, I don't think we want more opaque output like the RenderTree dump.

I am not against your plan if the readability of the tests can be good. I'd be happy to review patches toward testing dynamic changes in webpages.

Benjamin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130209/0470e42c/attachment.html>