[webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

[Dirk Pranke]

On Wed, Feb 13, 2013 at 11:33 PM, Benjamin Poulain <benjamin at webkit.org> wrote:
> On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke <dpranke at chromium.org> wrote:
>>
>> > Those changes are not harmless. There are people monitoring tests
>> > results
>> > full time in order to keep WebKit in good shape. No other part of WebKit
>> > require continuous attention.
>>
>> I'm sorry, but either I don't understand Dongsung's suggestion, or I
>> don't understand your criticism. What does this sentence paragraph
>> mean? Are you suggesting someone needs to look at this type of test
>> full time in a way that we don't have to look at the other tests?
>
>
> My lack of sleep is likely to blame here :)
>
> All I am saying is tests have a hight cost when they start failing, we
> should be careful how we design them so we can easily triage bad tests,
> flaky tests, and real errors.
>

No argument here :).

>
>> >> Case 1: CSS Filters & Shaders
>> >> I wanted this test functionality when I commented
>> >> http://webkit.org/b/97859#c19
>> >> If I want to make gaussian blur test, I prefer using 'getPixel' test as
>> >> follows,
>> >
>> >
>> > Why wasn't a ref-test a better solution in this particular case?
>> >
>>
>> I'm curious, what would you imagine the ref test contains?
>
>
> If I am not mistaken, the composition operations are parallel with the ones
> of SVG and Canvas (aren't they?).
> I would have attempted comparing the 3 implementations as it seems to me the
> pixels values should be the same.
>
> That was a genuine question about that case :)
>
>>
>>
>> >> Case 2: Fixed Position Element
>> >> [...]
>> >> function repeatedlyCalledDuringScrolling() {
>> >>     ASSERT(getPixel(15, 9) == white);
>> >>     ASSERT(getPixel(15, 10) == green);
>> >>     ASSERT(getPixel(9, 15) == white);
>> >>     ASSERT(getPixel(10, 15) == green);
>> >>     ....
>> >> }
>> >
>> >
>> > I think this shows what I said about correctness and readability:
>> > -Asserting the correctness of the test and the result becomes close to
>> > impossible for the reader. One has to review the full code to have a
>> > chance
>> > of understanding an error.
>> > -You cannot cover non trivial cases (images, text, form elements, etc).
>> > -And it is inefficient. You have to render each frame on the UIProcess,
>> > move
>> > it to the WebProcess, and box it for JavaScript to process (with pixel
>> > format conversions depending on your graphics system)
>> >
>> > Of the ideas raised, I think this is one of my least favorite for
>> > testing
>> > fixed positioning.
>> >
>>
>> Isn't his suggestion the equivalent of what we do today in text-only
>> tests? i.e., printing "pass" or "fail" and making you have to look at
>> the test itself to see what's being tested?
>>
>> If the correctness of the rendering depends on those 4 specific pixels
>> having those four specific values, how exactly are you going to verify
>> that by looking at it?
>>
>> Again, I think I'm just not understanding you here?
>
>
> When looking at a test test, you follow the flow to know what it is supposed
> to do and where it breaks.
>
> How are you supposed to know, _by reading the code_, that the color at
> position 15, 9 should be white?
>

is "the code" the test code, or the source code we're testing? Of
course, without any larger context, those assertions are impossible to
interpret. I figured he was just indicating what he meant by a
programmatic check, and a real test case would be more informative.

And of course it's easy to write bad tests that are neither readable
nor maintainable. But we also have lots of pixel tests that are
difficult for people not intimately familiar with the code being
tested to evaluate for correctness, too.

Lastly, I don't think he was suggesting that this would replace all
pixel tests; as you say, it is more well suited for some times of
tests than others.

As far as efficiency goes, even moving the picture back and forth
between two processes is still going to be far faster than running a
second reftest and comparing the output of the two pages.

-- Dirk