[webkit-dev] layout tests
David D. Kilzer
ddkilzer at kilzer.net
Fri Jan 19 05:31:53 PST 2007
On Jan 19, 2007, at 4:23 AM, Lars Knoll wrote:
> * run-webkit-tests generated results for new tests on the fly
> I've fixed this issue with r18976. run-webkit-tests does now not
> generate new
> results by default anymore. You'll have to pass the --new-tests
> flag to it to
> force it to do so.
What is the behavior on the buildbot if a test is committed without
results after applying this patch? It SHOULD fail! Currently, the
buildbot will generate new results (that automatically pass) with no
one the wiser.
> * All test results are stored together with the LayoutTests.
Thinking out loud, I like the idea of having separate results trees,
but I think it would be difficult to keep them in sync by putting
them in another repository, especially when committing. It will be
challenging enough to generate results for all the trees when a new
test is created or an existing test is fixed. Some example directory
structures (at the same level as LayoutTests):
Will we need some kind of a generate-test-results-on-all-ports-bot?
We can't expect every developer to have "one of each" kind of
system. Or must we expect a developer on each port to review new
tests and create updated test results on a per-port basis?
Would test results with the Qt port on the Mac be able to use the
test results with the Qt port on Linux (specifically, the image
results)? I could see subtle differences occurring between the same
"graphics port" on different operating systems.
Does Subversion have a way to do something like "check out this
entire tree, except for this directory" and then honor that
commitment when updating as well? Or would a custom update script be
needed, or a tool like svk?
It's too bad there isn't a way to store a set of base results, then
only store "expected differences" to each port. That would cut down
on the amount of space required by each new port's test results, but
it might be tricky to do with image results, and a text diff might be
as big or bigger than just new results.
Are there any other open source projects with multiple ports that
have already solved this problem?
Sorry...more questions than answers! :)
More information about the webkit-dev