[webkit-qt] Tests in Skipped List that Pass with Pixel Differences
Robert Hogan
lists at roberthogan.net
Mon Apr 19 14:41:14 PDT 2010
Looked into this a bit more:
Running the following, and looking at image_diffs in the output shows that
the completely wrong font is loaded:
WebKitTools/Scripts/run-webkit-tests --qt --platform mac --tolerance 2.0 -p
fast/block/basic/019.html
Running the following shows the font in WEBKIT_TESTFONTS getting selected:
export FC_DEBUG=1
WebKitBuild/unskip1/Release/bin/DumpRenderTree -v
LayoutTests/fast/block/basic/019.html
I don't understand enough about the fontconfig files in WEBKIT_TESTFONTS to
know how they should be changed to load a font that looks like the one
expected by Mac but presumably there must be something straightforward to
be done there.
Tor Arne, you seem to have set some of this up originally. Why does the
font loaded by the Qt DRT not resemble the one used by the Mac pixel tests?
robert at mwenge:~/Development/WebKit$
WebKitBuild/unskip1/Release/bin/DumpRenderTree -v
LayoutTests/fast/block/basic/019.html
FC_DEBUG=1
FC_DEBUG=1
Match Pattern has 15 elts (size 16)
family: "Arial"(s) "sans-serif"(w) "Helvetica"(w) "Nimbus Sans
L"(w) "sans-serif"(w)
slant: 0(i)(s) 0(i)(s)
weight: 100(i)(s) 100(i)(s)
width: 100(i)(s) 100(i)(s)
pixelsize: 16(f)(s) 16(f)(s)
hintstyle: 3(i)(s)
hinting: FcTrue(s)
verticallayout: FcFalse(s)
autohint: FcFalse(s)
globaladvance: FcTrue(s)
outline: FcTrue(s)
lang: "en-IE"(s)
fontversion: 2147483647(i)(s)
embeddedbitmap: FcTrue(s)
decorative: FcFalse(s)
Best score 0 0 100 100 3 0 0 0 0 2000 0 0 0 0 0 2.14748e+11Pattern has 15
elts (size 15)
family: "Nimbus Sans L"(w)
style: "Regular"(w)
slant: 0(i)(w)
weight: 80(i)(w)
width: 100(i)(w)
foundry: "urw"(w)
file: "/home/robert/Development/webkit/testfonts//n019003l.pfb"(w)
index: 0(i)(w)
outline: FcTrue(w)
scalable: FcTrue(w)
charset: 0000: 00000000 ffffffff ffffffff 7fffffff 00000000
ffffffff ffffffff ffffffff
0001: ffffffff ffffffff ffffffff ffffffff 00040000 00000000
00000000 00000000
0002: 0f000000 00000000 00000000 00000000 00000000 00000000
3f0002c0 00000000
0003: 00000000 00000000 00000000 00000000 00100000 10000000
00000000 00000000
0004: ffffffff ffffffff ffffffff 00000000 fffff000 ffffffff
ffff199f 033fffff
0020: 77180000 06010047 00000010 00000000 00000000 00001000
00000000 00000000
0021: 00400000 00000004 00000000 00000000 00000000 00000000
00000000 00000000
0022: 46260044 00000000 00000000 00000031 00000000 00000000
00000000 00000000
0025: 00000000 00000000 00000000 00000000 00000000 00000000
00000400 00000000
00f6: 00000000 00000000 00000000 00000000 00000000 00000000
00000008 00000000
00fb: 00000006 00000000 00000000 00000000 00000000 00000000
00000000 00000000
(w)
lang: aa|ab|af|ast|ava|ay|ba|be|bg|bi|br|bs|bua|ca|ce|ch|chm|co|cs|
cv|da|de|en|eo|es|et|eu|fi|fj|fo|fr|fur|fy|gd|gl|gv|ho|hr|hu|ia|id|ie|ik|
io|is|it|kaa|ki|kk|kl|ku|kum|kv|ky|la|lb|lez|lt|lv|mg|mh|mk|mo|mt|nb|nds|
nl|nn|no|nr|nso|ny|oc|om|os|pl|pt|rm|ro|ru|sah|se|sel|sh|sk|sl|sma|smj|smn|
so|sq|sr|ss|st|sv|sw|tg|tk|tn|tr|ts|tt|tyv|uk|uz|vo|vot|wa|wen|wo|xh|yap|
zu(w)
fontversion: 0(i)(w)
fontformat: "Type 1"(w)
decorative: FcFalse(w)
Match Pattern has 15 elts (size 16)
family: "Arial"(s) "sans-serif"(w) "Helvetica"(w) "Nimbus Sans
L"(w) "sans-serif"(w)
slant: 0(i)(s) 0(i)(s)
weight: 200(i)(s) 200(i)(s)
width: 100(i)(s) 100(i)(s)
pixelsize: 32(f)(s) 32(f)(s)
hintstyle: 3(i)(s)
hinting: FcTrue(s)
verticallayout: FcFalse(s)
autohint: FcFalse(s)
globaladvance: FcTrue(s)
outline: FcTrue(s)
lang: "en-IE"(s)
fontversion: 2147483647(i)(s)
embeddedbitmap: FcTrue(s)
decorative: FcFalse(s)
Best score 0 0 100 100 3 0 0 0 0 0 0 0 0 0 0 2.14748e+11Pattern has 15 elts
(size 15)
family: "Nimbus Sans L"(w)
style: "Bold"(w)
slant: 0(i)(w)
weight: 200(i)(w)
width: 100(i)(w)
foundry: "urw"(w)
file: "/home/robert/Development/webkit/testfonts//n019004l.pfb"(w)
index: 0(i)(w)
outline: FcTrue(w)
scalable: FcTrue(w)
charset: 0000: 00000000 ffffffff ffffffff 7fffffff 00000000
ffffffff ffffffff ffffffff
0001: ffffffff ffffffff ffffffff ffffffff 00040000 00000000
00000000 00000000
0002: 0f000000 00000000 00000000 00000000 00000000 00000000
3f0002c0 00000000
0003: 00000000 00000000 00000000 00000000 00100000 10000000
00000000 00000000
0004: ffffffff ffffffff ffffffff 00000000 fffff000 ffffffff
ffff199f 033fffff
0020: 77180000 06010047 00000010 00000000 00000000 00001000
00000000 00000000
0021: 00000000 00000004 00000000 00000000 00000000 00000000
00000000 00000000
0022: 06260044 00000000 00000000 00000031 00000000 00000000
00000000 00000000
0025: 00000000 00000000 00000000 00000000 00000000 00000000
00000400 00000000
00f6: 00000000 00000000 00000000 00000000 00000000 00000000
00000008 00000000
00fb: 00000006 00000000 00000000 00000000 00000000 00000000
00000000 00000000
(w)
lang: aa|ab|af|ast|ava|ay|ba|be|bg|bi|br|bs|bua|ca|ce|ch|chm|co|cs|
cv|da|de|en|eo|es|et|eu|fi|fj|fo|fr|fur|fy|gd|gl|gv|ho|hr|hu|ia|id|ie|ik|
io|is|it|kaa|ki|kk|kl|ku|kum|kv|ky|la|lb|lez|lt|lv|mg|mh|mk|mo|mt|nb|nds|
nl|nn|no|nr|nso|ny|oc|om|os|pl|pt|rm|ro|ru|sah|se|sel|sh|sk|sl|sma|smj|smn|
so|sq|sr|ss|st|sv|sw|tg|tk|tn|tr|ts|tt|tyv|uk|uz|vo|vot|wa|wen|wo|xh|yap|
zu(w)
fontversion: 0(i)(w)
fontformat: "Type 1"(w)
decorative: FcFalse(w)
Source:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd"><html lang="en-au"><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title></title>
<style>
h1 { border: 3px solid red; }
</style>
</head><body>
<p>These <h1>s should all be left-aligned:</p>
<h1 style="width: auto;">sample</h1>
<h1 style="width: 10em;">sample</h1>
<p>These <p>s should be (and are) left-aligned:</p>
<p style="width: auto;">sample</p>
<p style="width: 10em;">sample</p>
</body></html>
layer at (0,0) size 800x600
RenderView at (0,0) size 800x600
layer at (0,0) size 800x301
RenderBlock {HTML} at (0,0) size 800x301
RenderBody {BODY} at (8,16) size 784x269
RenderBlock {P} at (0,0) size 784x20
RenderText {#text} at (0,0) size 276x20
text run at (0,0) width 276: "These <h1>s should all be left-
aligned:"
RenderBlock {H1} at (0,41) size 784x47 [border: (3px solid #FF0000)]
RenderText {#text} at (3,3) size 111x41
text run at (3,3) width 111: "sample"
RenderBlock {H1} at (0,109) size 326x47 [border: (3px solid #FF0000)]
RenderText {#text} at (3,3) size 111x41
text run at (3,3) width 111: "sample"
RenderBlock {P} at (0,177) size 784x20
RenderText {#text} at (0,0) size 314x20
text run at (0,0) width 314: "These <p>s should be (and are)
left-aligned:"
RenderBlock {P} at (0,213) size 784x20
RenderText {#text} at (0,0) size 52x20
text run at (0,0) width 52: "sample"
RenderBlock {P} at (0,249) size 160x20
RenderText {#text} at (0,0) size 52x20
text run at (0,0) width 52: "sample"
#EOF
#EOF
#EOF
On Sunday 18 April 2010 23:35:49 Andras Becsi wrote:
> Hi Robert,
>
> this is a great plan.
> The only issue is that we are also facing this kind of pixel differences
> - between different releases of Qt: for example a Qt 4.6.2 build has
> ~300 failing tests on the bot which pass on a 4.6.1 build,
> - between different distributions: our buildbot runs Debian Lenny but
> developers frequently note that they have 300-500 different results on
> other distributions or other versions of the same distributions, and we
> also experience this problem on our ARMv5 tester, which might or might
> not be related to package version differences too.
>
> I'm strongly in favor of unskipping these tests if they are stable
> enougth (IIRC Ossy has a list somewhere), and rebaslineing the results,
> but we also have to take in account that if we do not find the cause of
> the differences between environments or it is not a bug of Qt (which I
> suppose) we have to update a whole lot of test results each time we
> switch Qt versions, and the landed results would only be relevant on our
> buildbot which would be an awkward thing. These tests would also need
> comparing of pixel results to the Mac ones because the DRT dumps are not
> always reliable. Pixel testing is a time consuming activity.
> Another partly relevant thing is that we were thinking about an tool,
> which would automatically run all the tests in the skipped list say once
> a week, to check if there are possible changes, but there are a few
> blocker tests which result in run-webkit-tests infinite loop. I took a
> note somewhere, I'll check and file a bug as a starting point. Fixing
> these would make some automation possible and ease the further actions
> on the skipped list.
>
> BR,
> Andras
>
> 2010-04-18 23:30 keltezéssel, Robert Hogan írta:
> > There is a definite class of tests in the Qt skipped list that fail
> > only because of unimportant pixel differences.
> >
> > I propose to open a master bug for these the purpose of which will be
> > to act as a staging pad for removing them from the Skipped list.
> >
> > I would see it working like this:
> >
> > - Identify a group of tests that only fail due to pixel differences
> > and for which it is clear pixel differences are not material
> > - Add a patch against LayoutTests/platform/qt/Skipped with the tests
> > removed.
> > - Along with the patch add a comment to bugzilla which will act as a a
> > 'manifest' listing each test to be removed and why it is safe to use
> > platform specific results. It might even be best to keep this manifest
> > as a file in the platform/qt tree.
> > - Send a note to the buildbot team requesting them to try the patch.
> > - Wait for a member of the buildbot team to run the patch against the
> > buildbot. The buildbot team can then post an updated patch with the
> > changes to the Skipped list and the platform specific results. They
> > mark it for review.
> > - A reviewer comes along, satisifies themselves that platform specific
> > results are justified for each of the skipped tests and approves.
> >
> > Keeping all this under a single bugzilla entry might get cumbersome
> > after a while, when that happens a new one can be created to replace
> > it and linked from the old one. I think it's important to have one
> > go-to place for this effort though.
> >
> > Thoughts? Do the buildbot team this is something that could work?
> > _______________________________________________
> > webkit-qt mailing list
> > webkit-qt at lists.webkit.org
> > http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt
>
> _______________________________________________
> webkit-qt mailing list
> webkit-qt at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-qt
>
-------------------------------------------------------
More information about the webkit-qt
mailing list