[webkit-dev] Adding New Layout Tests

[David D. Kilzer]

Reem Yazigi <reemyazigi at hotmail.com> wrote:

> Thanks for your helpful answers. I still have some more questions for you
guys:
> 
> 1- Dave said "Image test is required to capture expected results that a text
> based test can't". Can you please provide me with a path/name of one of these
> cases? 

$ ls LayoutTests/fast/images/*.html

> 2- I actually need to know which of the layout test cases I'm adding requires
> Image test, so I can generate the PNGs and Checksums for these specific test
> cases (through the -p option of the "run-webkit-tests" script)- Or do you
> think that it's better to generate the PNGs and checksums for all the test
cases
> I'm planning to add?

If you can write a test (that breaks before fixing the code) by writing a
text-only test case (see "layoutTestController.dumpAsText() in
LayoutTests/fast/css/computed-style.html), do that.  Image tests take a lot
longer to run and should be avoided unless the test requires it.

Otherwise, new test cases should have both text and pixel results if they're
not text-only tests.  (If you're porting to a new platform, I'd get the
text-only tests working first, then worry about the image tests later.)

> 3- How is the PNGs comparison implemented for regression testing? (in the
> text based tests it's  a string comparison how about the image based tests?)

Checksums of the two images are compared.  I take it you haven't looked through
the source of the run-webkit-tests script yet?

> 4- What are the Checksum files used for?

See above.

> 5- So, please correct me if there's anything missing in the following steps:
> In order to test a WebKit based port other than Safari and Qt with WebKit's
> Layout tests I need to:
>          a- update the "run-webkit-tests" Perl script.
>          b- port the "DumpRenderTree" app.
>          c- regenerate the expected results for each of the layout tests 
>           (knowing that the render tree dumps are different for each
platform)

Test results for a new platform should go under the LayoutTestResults directory
in a new "platform" directory.  Please make sure that the results are
repeatable, e.g., running the tests again after generating the results (both
individually and as a group) don't generate any differences unless the WebKit
code changes as well.

Good luck!

Dave