[webkit-dev] layout tests
lars at trolltech.com
Fri Jan 19 02:23:20 PST 2007
I have finally gotten the layout tests for the Qt build running a few days
ago. This does however bring two basic problems:
* run-webkit-tests generated results for new tests on the fly
This give problems both for the user and the build bots. As a user, you don't
see that there is a new test checked in for which you don't have a result.
This is especially a problem for the Qt build currently, but will also hit
the Mac build once we start checking in layout tests. A test that doesn't
have a result on a platform will most probably also need some manual
inspection to ensure it works correctly on that platform.
The build bots run into trouble, as they will generate/store the result for
the new test locally. Once someone commits the result, you'll end up getting
svn merging issues, as the files the build bot wants to check out already
exist on disk.
I've fixed this issue with r18976. run-webkit-tests does now not generate new
results by default anymore. You'll have to pass the --new-tests flag to it to
force it to do so.
* All test results are stored together with the LayoutTests.
This is ok for text only tests (as the results can be shared), only that
run-webkit-tests currently doesn't know whether a test is text only.
Once we submit the Qt test results (including pixel tests), you'll get 3 move
files per test case. In the long term we might get results from even more
platforms, completely cluttering the directories.
Checking out all the test results does already now take quite some time (and
people working on the Qt port don't really need the Mac results and vice
versa). Adding more results will at some point make this prohibitive (the
LayoutTests directory is at around 600MB currently, the number can be
multiplied with every platform that has tests running).
One way of solving this is to move all test results into a separate results
directory (or even separate repository). In there once could have
subdirectories shared (for the text only tests), mac and qt. We could also
add some sort of script to use instead of 'svn update' that will not check
out the results you don't need.
More information about the webkit-dev