[webkit-qt] Tests in Skipped List that Pass with Pixel Differences

[Robert Hogan]

There is a definite class of tests in the Qt skipped list that fail only 
because of unimportant pixel differences.

I propose to open a master bug for these the purpose of which will be to 
act as a staging pad for removing them from the Skipped list.

I would see it working like this:

- Identify a group of tests that only fail due to pixel differences and for 
which it is clear pixel differences are not material
- Add a patch against LayoutTests/platform/qt/Skipped with the tests 
removed.
- Along with the patch add a comment to bugzilla which will act as a a 
'manifest' listing each test to be removed and why it is safe to use 
platform specific results. It might even be best to keep this manifest as a 
file in the platform/qt tree.
- Send a note to the buildbot team requesting them to try the patch.
- Wait for a member of the buildbot team to run the patch against the 
buildbot. The buildbot team can then post an updated patch with the changes 
to the Skipped list and the platform specific results. They mark it for 
review.
- A reviewer comes along, satisifies themselves that platform specific 
results are justified for each of the skipped tests and approves.

Keeping all this under a single bugzilla entry might get cumbersome after a 
while, when that happens a new one can be created to replace it and linked 
from the old one. I think it's important to have one go-to place for this 
effort though.

Thoughts? Do the buildbot team this is something that could work?