[webkit-dev] WebKit unit test framework

[David Levin]

On Wed, Apr 20, 2011 at 3:00 PM, Timothy Hatcher <timothy at apple.com> wrote:

> I think having something the WebKit community owns and controls is
> preferred over importing and using a third-party library.
>
So that makes me prefer TestWebKitAPI (or something built from/on it) over
> gtest. And TestWebKitAPI already has a very simple test for WTF::Vector —
> just begging to be expanded.
>

I agree that this sounds attractive at first.  However, it feels like there
is a bigger picture.

There is a lot of work already done in gtest which would be nice to have (--
some of which will likely be necessary).

Here's a list right off the top of my head (from what I've seen of both):

   - TestWebKitAPI doesn't print out values when a test fails. (gtest has
   this support.)
   - I don't think there is any documentation for TestWebKitAPI (For, gtest
   there is http://code.google.com/p/googletest/wiki/Documentation).
   - I don't there are any test for the framework in TestWebKitAPI -- to be
   hackable in WebKit with confidence, this is needed. (gtest has extensive
   testing.)
   - TestWebKitAPI seems to only run one test at a time. When the ability is
   added to run all test, it would also be good to add the ability to run a set
   of test (gtest already has this).
   - In addition, gtest has a nice output (including a nice color output
   when supported by the terminal as well as output options which integrate
   better with automation -- see generating an xml report in
   http://code.google.com/p/googletest/wiki/AdvancedGuide).
   - The SetUp/TearDown functionality is a nice way isolate this type of
   stuff out of the way of tests.
   - In addition, the design of the api has gone through lots of discussion
   by concerned parties to work well. (This is more attention that we'd be able
   to expend on this.)

Someone could add these items to TestWebKitAPI eventually and some of these
items may never get done due to the cost/benefit ratio of doing them
for something just used in WebKit project.

In short, it seems to me that the effort to do any of this would be better
invested in other places where there isn't already something that works for
us.

fwiw, we could go with what we do with bugzilla where we start with gtest
and people change the code if needed.

dave


>
> On Apr 18, 2011, at 11:36 AM, David Levin wrote:
>
> *Issue: *There has been a long standing bug to add unit tests to WebKit (
> https://bugs.webkit.org/show_bug.cgi?id=21010). It was also mentioned<http://lists.macosforge.org/pipermail/webkit-dev/2009-January/006359.html>on webkit-dev that it would be helpful in various cases.
>
> *Landscape:* Surveying WebKit, it is looks like there are at least three
> testing frameworks being used: TestWebKitAPI/WebKitAPITest (in Tools),
> QTest, gtest (in Source/WebKit/chromium/). However, only one TestWebKitAPI
> has been used so far (as far as I can tell) for testing core WebKit items
> like WTF (though I was unaware of TestWebKitAPI until Friday).
>
> It seems like a good way to think about the issue of which to use in
> general in WebKit would be to decide on what would be desired in our
> framework and then see how each matches up.
>
> Here's my take on this. (It may be biased toward what I am familiar with
> but I welcome others to add their own criteria.)
>
> Criteria
>
> Musts:
>
>    - Compatible license with WebKit
>    - Builds/Can be built on the many platforms and build systems supported
>    by WebKit (ideally without extra installs).
>
> Useful:
>
>    - Easy to write tests
>    - Hackable to suit our needs
>    - Well tested features (to support hackability/stability)
>    - Supports filtering of tests so you can run just the test you care
>    about (and easily listing the tests).
>    - Supports writing out values when there is test failure. (For example,
>    if the is verifying that A == B but that is not true, then the values of A
>    and B should be printed.)
>    - Well documented
>
> thanks,
> dave
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
> — Timothy Hatcher
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110420/ac67476f/attachment.html>