[webkit-dev] Adding New Layout Tests

[David D. Kilzer]

> 1- Why not all of the layout tests are working on Qt? Aren't they supposed to
test
> the core?

Because the Qt port is still a work-in-progress, although it is progressing
rapidly!

> So, why do they all work on Safari but not on Qt?

Because they were written first for the Safari port.

> 2- Why is the render tree dumps different for each platform?

They should be the same, but invariably differences in fonts (and thus font
metrics) may cause the pages to be laid out slightly differently.  I'm not sure
if there are any other differences; in "theory", they should be the same for
all platforms.

> 3- Why would we generate PNGs and checksum files? Why would we use them and
for what?

Sometimes an "image test" is required to capture expected results that a
text-based test can't (such as rendering of graphic images in web pages with
proper colors).

You are correct in thinking that any test that does not need an image test
should be changed to a text-only test, but there are a number of original tests
where this was not done.  (Patches to fix this are welcome, however.)

> 4- How can we make sure the expected results files that get automatically
generated
> for new tests are correct?

Visual inspection by a human.  If this is a new port, you should probably write
some kind of script that compares "known good" results with a new port's
results using some kind of "confidence interval" logic.  I think the Qt guys
may have written something like this already for their port?  I don't see it in
WebKitTools/Scripts, though.

Dave


Reem Yazigi <reemyazigi at hotmail.com> wrote:

> Dave, Kirby and all,
> 
> Thanks for the useful answers and documentation. I have some more questions
that I
> need answers for before I start adding new test cases:
> 
> 1- Why not all of the layout tests are working on Qt? Aren't they supposed to
test
> the core? So, why do they all work on Safari but not on Qt?
> 
> 2- Why is the render tree dumps different for each platform?
> 
> 3- Why would we generate PNGs and checksum files? Why would we use them and
for what?
> 
> 4- How can we make sure the expected results files that get automatically
generated
> for new tests are correct?
> 
> Thanks,
> 
> Reem.
> 
> "David D. Kilzer" <ddkilzer at kilzer.net> wrote:
> > > Reem Yazigi <reemyazigi at hotmail.com> wrote:
> > > 
> > >  > 1- Are the expected-results files browser dependent?
> > > 
> > > The expected results files are simply text, but there are two basic
types:
> > > render tree dumps and text dumps.  The render tree dumps will be
different for
> > > each platform, while the text dumps should be the same (or much closer).
> > > 
> > >  > (are the ones "run-webkit-tests" creates for Qt identical
> > >  > to the ones it creates for Safari?
> > > 
> > > No.
> > > 
> > >  > Do I have to change that if I want to run the tests for gdk?)
> > > 
> > > Yes, but run-webkit-tests has a way to save results in a different
directory
> > > structure, which is what Qt is doing.
> > > 
> > >  > 2- When I run the "run-webkit-tests" for Qt, 238 test cases
> > >  > get skipped and only about 4100 get executed! I believe they're
> > >  > about 7700 on Safari!! Why is it that different?
> > > 
> > > Because not all of the tests are working on Qt yet.
> > > 
> > >  > 3- Is there any parameter that needs to be passed to the
> > >  > "run-webkit-tests" script or to the "DumpRenderTree" in order
> > >  > to specify the browser's window dimensions?? I'm actually getting
> > >  > different results when I run "run-webkit-tests" for Qt on
> > >  > different devices!!
> > > 
> > > I don't know about Qt, but the dimensions are hard-coded in the
> > > DumpRenderTree.m source file for the native Mac version (800x600).
> > > 
> > >  > I couldn't access the IRC channel!! Is it still available?
> > > 
> > > The #webkit channel on irc.freenode.net?  http://webkit.org/contact.html
> > > 
> > > Dave