[webkit-dev] layout tests

[Lars Knoll]

Hi,

I have finally gotten the layout tests for the Qt build running a few days 
ago. This does however bring two basic problems:

* run-webkit-tests generated results for new tests on the fly

This give problems both for the user and the build bots. As a user, you don't 
see that there is a new test checked in for which you don't have a result. 
This is especially a problem for the Qt build currently, but will also hit 
the Mac build once we start checking in layout tests. A test that doesn't 
have a result on a platform will most probably also need some manual 
inspection to ensure it works correctly on that platform.

The build bots run into trouble, as they will generate/store the result for 
the new test locally. Once someone commits the result, you'll end up getting 
svn merging issues, as the files the build bot wants to check out already 
exist on disk.

I've fixed this issue with r18976. run-webkit-tests does now not generate new 
results by default anymore. You'll have to pass the --new-tests flag to it to 
force it to do so.

* All test results are stored together with the LayoutTests. 

This is ok for text only tests (as the results can be shared), only that 
run-webkit-tests currently doesn't know whether a test is text only. 

Once we submit the Qt test results (including pixel tests), you'll get 3 move 
files per test case. In the long term we might get results from even more 
platforms, completely cluttering the directories.

Checking out all the test results does already now take quite some time (and 
people working on the Qt port don't really need the Mac results and vice 
versa). Adding more results will at some point make this prohibitive (the 
LayoutTests directory is at around 600MB currently, the number can be 
multiplied with every platform that has tests running).

One way of solving this is to move all test results into a separate results 
directory (or even separate repository). In there once could have 
subdirectories shared (for the text only tests), mac and qt. We could also 
add some sort of script to use instead of 'svn update' that will not check 
out the results you don't need.


Opinions?

Cheers,
Lars