Platform improvements to the layout tests
Hi everybody, We've recently made some changes to the layout tests that allow us to have much better flexibility in the way we handle platforms. Most of my examples use Leopard only to demonstrate the hierarchical nature of the new tests and apply equally to all platforms. Test results are now searched for using a platform hierarchy defined within the script for your platform. If you were running Leopard for example, when run-webkit-tests comes across the css1/basic/ inheritance.html test, it will first search platform/mac-leopard/css1/ basic for the results, then platform/mac/css1/basic and finally in the same directory as the test. As for new tests, if the test produces text-based results (ie: dom/), it will still have it's results generated alongside the test. Pixel and render-tree test results will now be generated in the test's most common corresponding platform/ directory. So creating a new test in fast/backgrounds on Leopard will have the results generated in platform/mac/fast/backgrounds. If that test's results are specific to Leopard, they will manually have to be moved to the mac-leopard directory and a different set of results created on Tiger (or the test could just be made Leopard-only). The mac pixel test results are still in their old locations, but I'll be migrating those to platform/mac as time goes on. run-webkit-tests also now allows for platform-specific tests and Skipped files as well. The addition of the former should hopefully cut down on the number of entries in the latter. For those working on new ports, setting up all of this for your port is really easy. Once you have DRT up and running, you only need to create your directory (or directories) in platform/ and define the $platform variable in run-webkit-tests to point to it. Hopefully this clears up any confusion anyone may have. Happy hacking! - Matt
Thanks for the explanation, Matt! For those who haven't been following the evolution of run-webkit- tests as closely as Matt and I have, the point of the changes to run- webkit-tests was to allow the separation of cross-platform and platform-specific tests. Currently there are a lot of tests sitting under LayoutTests that are Mac-only (such as fast/AppleScript), and there are *loads* of test results that are Mac/CG-only due to font metrics (basically all the render tree dumps). This has made things difficult as other platforms have tried to get the regression tests running, so the hope is that we can now start separating the platform- specific tests from the truly cross-platform ones, and ditto for the expected results. A full description of the changes to run-webkit-tests can be found at <http://trac.webkit.org/projects/webkit/wiki/RunWebKitTestsDesign>. On Aug 14, 2007, at 9:52 PM, Matt Lilek wrote:
Pixel and render-tree test results will now be generated in the test's most common corresponding platform/ directory. So creating a new test in fast/backgrounds on Leopard will have the results generated in platform/mac/fast/backgrounds. If that test's results are specific to Leopard, they will manually have to be moved to the mac-leopard directory and a different set of results created on Tiger (or the test could just be made Leopard-only).
The basic question you should ask yourself when deciding where to place a new test is "Does this test apply to all platforms, or just mine?" Tests meant to highlight a bug that occurred on a specific platform can still be made cross-platform tests if the expected behavior is valid on all platforms. And it's OK for a cross-platform test to have platform-specific results -- as I said above, render tree dumps are currently not easily shared between platforms (though this is definitely something we'd like to improve!). -Adam
participants (2)
-
Adam Roben
-
Matt Lilek