[webkit-qt] About the fonts used for QtWebKit tests

[Osztrogonac Csaba]

Hi,

Balazs Kelemen írta:

[snip]

>> Ossy, another issue: currently we have some outdated *-expected.png
>> files hanging around the platform/qt. I say outdated because I don't
>> think they pass anymore based on my tests, the history, and the fact
>> that we don't do pixel tests. What's the approach here: do we generate
>> new versions of the ones that exist again? We get rid of them until we
>> have a bot running pixel tests? In the IRC people mentioned they are
>> good for reference purposes, but I think the other platforms updated
>> results could serve better as a reference than this.
> 
> I agree on that. It doesn't make sense to store outdated png's in the 
> tree. That just confuses people.

I have an objection against removing "outdated" png files from trunk.
They aren't so outdated as you think it. :) If a pixel test fails,
it doesn't mean that the png result is absolutely invalid.

When we update an *-expected.txt, we update its *-expected.png too. (almost always)
Of course/unfortunately not everybody updates png files, but we can live with
this small noise. (And we can catch this cases easily by scripts.) Until the test
isn't disabled, *-expected.txt guarantees that the results doesn't change significantly.
Of course minimal pixel differences can occure which can't be catched by text rendertree
dump. But the png is still a good reference if we have to update the test result once.
It's much harder to compare actual pngs to Mac/GTK/Chromium pngs (which have different
fonts, different scrollbars, input boxes, linebreaks, ...) than 1-2 month old Qt pngs.
Just try to run ORWT --platform <mac/qt> -p <css2.1/fast> and you will see the non-trivial
differences. So outdated png files shouldn't confuse anybody, but they are very helpful for
gardeners (Szeged guys). Otherwise now only Chromium bots run pixel tests, but GTK/Apple
and us regularly check in pngs. Of course it would be great if we can run pixel tests on
the bots ... but pixel results aren't so stable (can be different on 32/64 bit, release/
debug, Qt4.8/Qt5, ..., can depends on DRT sideefects, ...) and keep them up-to-date/
keep the tree green would need more and more full time gardener.

But we should be optimistic. ;-) With updating ~5000 test results (and their pngs too)
we will have more or less up-to-date pixel results. So we can try to start run pixel
tests only on one bot to experiment a little bit. You'll see that maintaining pixel
results aren't so easy.

br,
Ossy