[webkit-dev] Can we distinguish imported tests in LayoutTests/css3 ?

[Darin Adler]

On Jun 11, 2012, at 7:33 PM, Mike Lawther wrote:

> Does having the 'fast' directory still serve a useful purpose?

Not sure it does.

The original intent was to put more extensive complete tests that were slow but had greater coverage outside the “fast” directory, so that you could run the majority of the most useful tests in a few seconds.

Times to run tests have increased so much, despite the parallel test runner, that it’s no longer a useful distinction.

A new logical organization for tests would be welcome, and it’s a big project where I am not sure we have a clear enough plan to start on it incrementally. Here are a few considerations:

- Moving test files around could cause some huge churn, with some super-slow check-outs for everyone working on the project and many old patches potentially needing to be rebased. This is especially true when image results are involved.

- Each time we change the tests we need to make sure we haven’t broken new tests on various platforms and that we’ve updated the skipped or test expectations files properly.

- Calling the entire directory that hosts our regression test suite LayoutTests seems wrong; that name was apropos when most of the tests were for layout issues, very early in the life of the test machinery.

- Imported test suites ought to be gathered together in a single directory then grouped by where they were imported from by having a directory for each source.

- We want to continue the practice of keeping copies imported test suites as close to the original as possible. In some cases in the past that meant preserving incorrect imported tests with expected “failures” that weren’t really failures. Often we’d put corrected copies elsewhere in the test tree while waiting for or working to get the imported test to fixed at the source.

- Keeping the tests that require the HTTP server in a separate directory still seems like it might serve a purpose, although there seems to be an extra directory level in the test hierarchy there that is not helpful.

- When rearranging tests that share the test shell JavaScript files we’d likely need to update relative paths. Might also be nice to put that test shell in a more logical location.

- Making JavaScript-only tests runnable without WebKit’s involvement would be a welcome feature; great for anyone working on JavaScriptCore. That means we’d want to be sure to put those into their own directories. Ideally that would eventually be coupled by a project to move the Mozilla test suite currently in JavaScriptCore into the main WebKit regression tests.

-- Darin