[webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

[noam.rosenthal at nokia.com]

From: ext Benjamin Poulain <benjamin at webkit.org<mailto:benjamin at webkit.org>>
Date: Thursday, February 14, 2013 8:07 AM
To: Dongsung Huang <luxtella at company100.net<mailto:luxtella at company100.net>>
Cc: "webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>" <webkit-dev at lists.webkit.org<mailto:webkit-dev at lists.webkit.org>>
Subject: Re: [webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

On Wed, Feb 13, 2013 at 9:38 PM, Dongsung Huang <luxtella at company100.net<mailto:luxtella at company100.net>> wrote:
I like this idea. I cannot find any harm if we have this functionality.

Those changes are not harmless. There are people monitoring tests results full time in order to keep WebKit in good shape. No other part of WebKit require continuous attention.

Case 1: CSS Filters & Shaders
I wanted this test functionality when I commented http://webkit.org/b/97859#c19
If I want to make gaussian blur test, I prefer using 'getPixel' test as follows,

Why wasn't a ref-test a better solution in this particular case?
Because gaussian blurs on the GPU are not accurate, and look slightly different on different GPUs, but usually "close enough".
We need a way to measure "close enough" for features where all you can get is "close enough"! Ref-tests and pixel-tests are way to rigid for that, and require constant rebaselines and headaches.


Case 2: Fixed Position Element
[...]
function repeatedlyCalledDuringScrolling() {
    ASSERT(getPixel(15, 9) == white);
    ASSERT(getPixel(15, 10) == green);
    ASSERT(getPixel(9, 15) == white);
    ASSERT(getPixel(10, 15) == green);
    ....
}

I think this shows what I said about correctness and readability:
-Asserting the correctness of the test and the result becomes close to impossible for the reader. One has to review the full code to have a chance of understanding an error.
We can generate PNGs when a snapshot is requested so that the tester can eyeball the results.

-You cannot cover non trivial cases (images, text, form elements, etc).
-And it is inefficient. You have to render each frame on the UIProcess, move it to the WebProcess, and box it for JavaScript to process (with pixel format conversions depending on your graphics system)
Another option is to do expose a partial comparison mechanism that runs in  the UI process.
E.g.
var snapshotHandle = testRunner.createSnapshot();
testRunner.comparePixel(snapshotHandle, 15, 20, 'white', { tolerance: 0.2 });

This way we can generate a visual representation of failures.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130214/2621950d/attachment.html>