[webkit-dev] Testing feature suggestion: animation/interaction pixel-results "on the fly"

[Benjamin Poulain]

On Wed, Feb 13, 2013 at 11:16 PM, Dirk Pranke <dpranke at chromium.org> wrote:

> > Those changes are not harmless. There are people monitoring tests results
> > full time in order to keep WebKit in good shape. No other part of WebKit
> > require continuous attention.
>
> I'm sorry, but either I don't understand Dongsung's suggestion, or I
> don't understand your criticism. What does this sentence paragraph
> mean? Are you suggesting someone needs to look at this type of test
> full time in a way that we don't have to look at the other tests?
>

My lack of sleep is likely to blame here :)

All I am saying is tests have a hight cost when they start failing, we
should be careful how we design them so we can easily triage bad tests,
flaky tests, and real errors.


>> Case 1: CSS Filters & Shaders
> >> I wanted this test functionality when I commented
> >> http://webkit.org/b/97859#c19
> >> If I want to make gaussian blur test, I prefer using 'getPixel' test as
> >> follows,
> >
> >
> > Why wasn't a ref-test a better solution in this particular case?
> >
>
> I'm curious, what would you imagine the ref test contains?
>

If I am not mistaken, the composition operations are parallel with the ones
of SVG and Canvas (aren't they?).
I would have attempted comparing the 3 implementations as it seems to me
the pixels values should be the same.

That was a genuine question about that case :)


>
> >> Case 2: Fixed Position Element
> >> [...]
> >> function repeatedlyCalledDuringScrolling() {
> >>     ASSERT(getPixel(15, 9) == white);
> >>     ASSERT(getPixel(15, 10) == green);
> >>     ASSERT(getPixel(9, 15) == white);
> >>     ASSERT(getPixel(10, 15) == green);
> >>     ....
> >> }
> >
> >
> > I think this shows what I said about correctness and readability:
> > -Asserting the correctness of the test and the result becomes close to
> > impossible for the reader. One has to review the full code to have a
> chance
> > of understanding an error.
> > -You cannot cover non trivial cases (images, text, form elements, etc).
> > -And it is inefficient. You have to render each frame on the UIProcess,
> move
> > it to the WebProcess, and box it for JavaScript to process (with pixel
> > format conversions depending on your graphics system)
> >
> > Of the ideas raised, I think this is one of my least favorite for testing
> > fixed positioning.
> >
>
> Isn't his suggestion the equivalent of what we do today in text-only
> tests? i.e., printing "pass" or "fail" and making you have to look at
> the test itself to see what's being tested?
>
> If the correctness of the rendering depends on those 4 specific pixels
> having those four specific values, how exactly are you going to verify
> that by looking at it?
>
> Again, I think I'm just not understanding you here?
>

When looking at a test test, you follow the flow to know what it is
supposed to do and where it breaks.

How are you supposed to know, _by reading the code_, that the color at
position 15, 9 should be white?

Benjamin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20130213/374ff9d1/attachment.html>