[webkit-qt] Tests in Skipped List that Pass with Pixel Differences

[Andras Becsi]

Hi Robert,

this is a great plan.
The only issue is that we are also facing this kind of pixel differences
  - between different releases of Qt: for example a Qt 4.6.2 build has 
~300 failing tests on the bot which pass on a 4.6.1 build,
  - between different distributions: our buildbot runs Debian Lenny but 
developers frequently note that they have 300-500 different results on 
other distributions or other versions of the same distributions, and we 
also experience this problem on our ARMv5 tester, which might or might 
not be related to package version differences too.

I'm strongly in favor of unskipping these tests if they are stable 
enougth (IIRC Ossy has a list somewhere), and rebaslineing the results, 
but we also have to take in account that if we do not find the cause of 
the differences between environments or it is not a bug of Qt (which I 
suppose) we have to update a whole lot of test results each time we 
switch Qt versions, and the landed results would only be relevant on our 
buildbot which would be an awkward thing. These tests would also need 
comparing of pixel results to the Mac ones because the DRT dumps are not 
always reliable. Pixel testing is a time consuming activity.
Another partly relevant thing is that we were thinking about an tool, 
which would automatically run all the tests in the skipped list say once 
a week, to check if there are possible changes, but there are a few 
blocker tests which result in run-webkit-tests infinite loop. I took a 
note somewhere, I'll check and file a bug as a starting point. Fixing 
these would make some automation possible and ease the further actions 
on the skipped list.

BR,
Andras

2010-04-18 23:30 keltezéssel, Robert Hogan írta:
> There is a definite class of tests in the Qt skipped list that fail only
> because of unimportant pixel differences.
>
> I propose to open a master bug for these the purpose of which will be to
> act as a staging pad for removing them from the Skipped list.
>
> I would see it working like this:
>
> - Identify a group of tests that only fail due to pixel differences and for
> which it is clear pixel differences are not material
> - Add a patch against LayoutTests/platform/qt/Skipped with the tests
> removed.
> - Along with the patch add a comment to bugzilla which will act as a a
> 'manifest' listing each test to be removed and why it is safe to use
> platform specific results. It might even be best to keep this manifest as a
> file in the platform/qt tree.
> - Send a note to the buildbot team requesting them to try the patch.
> - Wait for a member of the buildbot team to run the patch against the
> buildbot. The buildbot team can then post an updated patch with the changes
> to the Skipped list and the platform specific results. They mark it for
> review.
> - A reviewer comes along, satisifies themselves that platform specific
> results are justified for each of the skipped tests and approves.
>
> Keeping all this under a single bugzilla entry might get cumbersome after a
> while, when that happens a new one can be created to replace it and linked
> from the old one. I think it's important to have one go-to place for this
> effort though.
>
> Thoughts? Do the buildbot team this is something that could work?
> _______________________________________________
> webkit-qt mailing list
> webkit-qt at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt