[webkit-dev] Platform improvements to the layout tests

[Adam Roben]

    Thanks for the explanation, Matt!

    For those who haven't been following the evolution of run-webkit- 
tests as closely as Matt and I have, the point of the changes to run- 
webkit-tests was to allow the separation of cross-platform and  
platform-specific tests. Currently there are a lot of tests sitting  
under LayoutTests that are Mac-only (such as fast/AppleScript), and  
there are *loads* of test results that are Mac/CG-only due to font  
metrics (basically all the render tree dumps). This has made things  
difficult as other platforms have tried to get the regression tests  
running, so the hope is that we can now start separating the platform- 
specific tests from the truly cross-platform ones, and ditto for the  
expected results.

    A full description of the changes to run-webkit-tests can be found  
at <http://trac.webkit.org/projects/webkit/wiki/RunWebKitTestsDesign>.

On Aug 14, 2007, at 9:52 PM, Matt Lilek wrote:

> Pixel and render-tree test results will now be generated in the  
> test's most common corresponding platform/ directory.  So creating a  
> new test in fast/backgrounds on Leopard will have the results  
> generated in platform/mac/fast/backgrounds.  If that test's results  
> are specific to Leopard, they will manually have to be moved to the  
> mac-leopard directory and a different set of results created on  
> Tiger (or the test could just be made Leopard-only).

    The basic question you should ask yourself when deciding where to  
place a new test is "Does this test apply to all platforms, or just  
mine?" Tests meant to highlight a bug that occurred on a specific  
platform can still be made cross-platform tests if the expected  
behavior is valid on all platforms. And it's OK for a cross-platform  
test to have platform-specific results -- as I said above, render tree  
dumps are currently not easily shared between platforms (though this  
is definitely something we'd like to improve!).

-Adam