[webkit-dev] Using ref tests for repaint bugs

[Andrei Bucur]

I want to give everyone an update on how things are evolving. For now,
I've focused most of the development on the Mac platform.

For the ref-tests-as-repaint-tests path, I've added an optional parameter
to LTC.display() that should control if the repaint rectangles are tracked
or not. If not specified or true, the repaint rectangles are tracked.
After running very basic experiments, this seems to detect if repaint
artifacts are present in a test.

For the repaint-rects-as-repaint-tests path, I have some questions about
what should be the best testing methodology. I'm thinking about something
along these lines:

* I don't think a new script option or DRT parameter are necessary; the
repaint rectangles for a test can be extracted together with the pixel
results
* The format for the rectangles data should be simple, maybe X,Y,W,H
separated by new-lines? Or the smallest rectangle that includes all of the
repaint rectangles? Or another suggestion?
* When initially running a repaint test, both a PNG and RR
(repaint-rectangles) files are generated as baselines. At subsequent
executions of the test, the RR expected output is preferred over the pixel
results. This means that if the test author modifies the expected PNG, the
test should still pass because the RR baseline exists. However, if a pixel
test in preferred, deleting the expected RR output will force the test
runner to fallback on the classic image diff code path.
* A port may chose to repaint everything, even for partial canvas
invalidation. Do you think it's correct to evaluate a test as passed if
the expected repaint rectangles are included in the actual repaint
rectangles? The simplest and straightforward way to compare the actual
results with the expected ones is a simple bytes diff, but I'm not
convinced it's correct. Any thoughts on this?

The chromium DRT project has a different strategy for masking with the
repaint rectangles; could someone help me understand better what happens
in the WebViewHost::paintInvalidatedRegion() method and why exactly is
iterating 3 times through that code necessary? (the comment looks a little
obsoleted and I'd appreciate some clarifications).

Overall, I see some difficulties in keeping the repaint rectangles based
testing  to be platform independent. Do you have any suggestions that may
help with this? Once I'm finished patching the Python scripts to support
the RR tests, I'll upload a patch so people can look over the approach.

Thx,
Andrei.