On Jan 19, 2007, at 4:23 AM, Lars Knoll wrote:
* run-webkit-tests generated results for new tests on the fly
[...]
I've fixed this issue with r18976. run-webkit-tests does now not generate new results by default anymore. You'll have to pass the --new-tests flag to it to force it to do so.
What is the behavior on the buildbot if a test is committed without results after applying this patch? It SHOULD fail! Currently, the buildbot will generate new results (that automatically pass) with no one the wiser.
* All test results are stored together with the LayoutTests.
Thinking out loud, I like the idea of having separate results trees, but I think it would be difficult to keep them in sync by putting them in another repository, especially when committing. It will be challenging enough to generate results for all the trees when a new test is created or an existing test is fixed. Some example directory structures (at the same level as LayoutTests): LayoutTestsResultsMac LayoutTestsResultsQt LayoutTestsTextResults LayoutTestsImageResults/mac LayoutTestsImageResults/qt LayoutTestsResults/text LayoutTestsResults/image/mac LayoutTestsResults/image/qt Will we need some kind of a generate-test-results-on-all-ports-bot? We can't expect every developer to have "one of each" kind of system. Or must we expect a developer on each port to review new tests and create updated test results on a per-port basis? Would test results with the Qt port on the Mac be able to use the test results with the Qt port on Linux (specifically, the image results)? I could see subtle differences occurring between the same "graphics port" on different operating systems. Does Subversion have a way to do something like "check out this entire tree, except for this directory" and then honor that commitment when updating as well? Or would a custom update script be needed, or a tool like svk? It's too bad there isn't a way to store a set of base results, then only store "expected differences" to each port. That would cut down on the amount of space required by each new port's test results, but it might be tricky to do with image results, and a text diff might be as big or bigger than just new results. Are there any other open source projects with multiple ports that have already solved this problem? Sorry...more questions than answers! :) Dave