[webkit-dev] testharness Wiki page added

[Jacob Goldstein]

And now with the attachment I mentioned in my previous message



On 6/1/12 2:40 PM, "Jacob Goldstein" <jacobg at adobe.com<mailto:jacobg at adobe.com>> wrote:


On 5/31/12 7:55 PM, "Maciej Stachowiak" <mjs at apple.com<mailto:mjs at apple.com>> wrote:


On May 31, 2012, at 5:51 PM, Jacob Goldstein <jacobg at adobe.com<mailto:jacobg at adobe.com>> wrote:

I haven't found that to be the case for the tests I have written for each suite, the output from testharness can be as simple as "PASS" or "FAIL", or include additional debug information defined by the test author.  That being said, my experience is likely more limited than yours.  Peter Linss, who wrote and maintains the W3C testharness, would likely be open to suggestions for improvement if there is anything specific that we want to change.

The source is more verbose. The output is uglier and less clear, at least by default. "PASS" and "FAIL" is not as good as what our test driver does by default, because you can't as easily tell what passed or failed or why. Our own harness tells you the expression the test evaluated that failed, and what the expected result was, so it's much quicker to debug a failure.

I have given my feedback to James Graham before, including showing him how our test harness works. At the time, he was not open to making changes, in part because of details of Opera's internal testing setup and in part because "eval is evil". I really would rather not reduce all our DOM tests to the Opera-driven level of source legibility and output quality.

Let me give you an example. This zip file contains an actual w3c test case, and a webkit-style conversion of the same test:


I understand your concern, however the example you attached doesn't use the W3C testharness to make assertions.  Instead, it is using a custom assertEquals method from the file domtestcase.js.  The output from this example appears to be totally defined by the test author.

In the attached zip I added a new file, anchor_href-w3c_EDITED.html, that uses testharness and shows the results from its assert_equals method.  I think these results are more inline with what you want – i.e. the default results include both the actual and expected values for the assertion that failed.  If you launch this file, the output from testharness appears at the bottom of the page.  The rest of the content is from the original test file.

For example, the output from the failing test, using testharness.js, is as follows:

FAIL Test that a href attribute contains search string assert_equals: Optional debug info. expected "http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.x" but got "http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter"

Two portions of the output above is defined by the test author and so can be customized or omitted altogether.  The test name or description, i.e. "Test that a href attribute contains search string" , and the debug message "Optional debug info.".  These fields can be made to show anything, including the expression being evaluated, if so desired.

By default, testharness outputs "FAIL" and the actual and expected value for any assertion that fails.

Compare this to the output from your example using js-test-pre.js:

FAIL document.getElementById('anchor_5').getAttribute('href') should be http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.x. Was http://www.anothersite.com/path/page.htm?parameter=this%20is%a%20parameter.


And the two look much more similar.  That is not to say that testharness couldn't benefit from some tweaks, but I think the results are closer to what we already have than demonstrated by your example.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120601/64f2c5fc/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test-example.zip
Type: application/zip
Size: 64523 bytes
Desc: test-example.zip
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20120601/64f2c5fc/attachment-0001.zip>