[webkit-dev] layout tests

[David D. Kilzer]

On Jan 19, 2007, at 4:23 AM, Lars Knoll wrote:

> * run-webkit-tests generated results for new tests on the fly
>
> [...]
>
> I've fixed this issue with r18976. run-webkit-tests does now not  
> generate new
> results by default anymore. You'll have to pass the --new-tests  
> flag to it to
> force it to do so.

What is the behavior on the buildbot if a test is committed without  
results after applying this patch?  It SHOULD fail!  Currently, the  
buildbot will generate new results (that automatically pass) with no  
one the wiser.

> * All test results are stored together with the LayoutTests.

Thinking out loud, I like the idea of having separate results trees,  
but I think it would be difficult to keep them in sync by putting  
them in another repository, especially when committing.  It will be  
challenging enough to generate results for all the trees when a new  
test is created or an existing test is fixed.  Some example directory  
structures (at the same level as LayoutTests):

   LayoutTestsResultsMac
   LayoutTestsResultsQt

   LayoutTestsTextResults
   LayoutTestsImageResults/mac
   LayoutTestsImageResults/qt

   LayoutTestsResults/text
   LayoutTestsResults/image/mac
   LayoutTestsResults/image/qt

Will we need some kind of a generate-test-results-on-all-ports-bot?   
We can't expect every developer to have "one of each" kind of  
system.  Or must we expect a developer on each port to review new  
tests and create updated test results on a per-port basis?

Would test results with the Qt port on the Mac be able to use the  
test results with the Qt port on Linux (specifically, the image  
results)?  I could see subtle differences occurring between the same  
"graphics port" on different operating systems.

Does Subversion have a way to do something like "check out this  
entire tree, except for this directory" and then honor that  
commitment when updating as well?  Or would a custom update script be  
needed, or a tool like svk?

It's too bad there isn't a way to store a set of base results, then  
only store "expected differences" to each port.  That would cut down  
on the amount of space required by each new port's test results, but  
it might be tricky to do with image results, and a text diff might be  
as big or bigger than just new results.

Are there any other open source projects with multiple ports that  
have already solved this problem?

Sorry...more questions than answers!  :)

Dave