[webkit-dev] LayoutTests take too long to run

[Ojan Vafai]

SUMMARY
I doubt there's much room for improvement in the test harness itself outside
of running tests in parallel. However, there's clearly a lot to be gained by
making individual tests faster. For Chromium runs, the median time to run a
test is ~16ms. It's not apples to apples, but I would be surprised if there
were a significant difference between this and WebKit's run-webkit-tests.

With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the
whole test suite.

DETAILS
For the Chromium run_webkit_test.py, here's the stats in seconds from a
TestShell run on Windows Debug. FWIW, we spit these stats out for every run.
The eventual goal was to track this in a set of graphs, but I never got
around to implementing that.

Median: 0.0160000324249
Mean: 0.163495878646
90th percentile: 0.077999830246
99th percentile: 3.1099998951
Standard deviation: 0.519017629449

Here's the times for the five slowest top-level directories:
 LayoutTests\editing took 43.5 seconds to run 976 tests.
LayoutTests\svg took 53.7 seconds to run 891 tests.
LayoutTests\fast took 378.7 seconds to run 3695 tests.
LayoutTests\http took 396.9 seconds to run 632 tests.
LayoutTests\media took 596.3 seconds to run 84 tests.

This might be Chromium specific, but I'm seeing many of the the media layout
tests taking 8-10 seconds.

Ojan

On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <eric at webkit.org> wrote:

> Yeah, I stared a slowest run before going to bed, and got the following:
> The 10 slowest tests:
>
> 9.58 secs: editing/selection/move-left-right.html
> 9.37 secs: fast/js/array-filter.html
> 7.71 secs: editing/selection/extend-selection.html
> 6.82 secs: fast/js/sort-randomly.html
> 6.25 secs: http/tests/cache/subresource-expiration.html
> 5.77 secs: fast/js/array-enumerators-functions.html
> 5.41 secs: fast/events/click-count.html
> 5.39 secs: fast/js/toString-and-valueOf-override.html
> 5.26 secs: fast/js/try-catch-crash.html
> 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
>
> that's 10% of our time right there. :)  I bet some of those have 5 second
> timers and should be easy to fix.  I'll take a look, anyone else who would
> like to help would be most welcome. :)
>
> I remember Ojan finding that running tests in parallel was the biggest
> overall win, but there is definitely a lot of low-hanging fruit in the
> slowest tests too.  I'm curious what the fastest tests run in.  I would
> expect the harness cost to be near-zero, but I'm not sure.  (i.e. what's the
> runtime of run-webkit-tests with 10000 empty test.  I would expect only a
> few seconds.)
>
> -eric
>
> On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>
>>
>> On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
>>
>>  681.70s total testing time
>>>
>>> That's 11.5 minutes for every patch I want to land.  (because I run the
>>> layout tests before landing anything, as part of bugzilla-tool).
>>>
>>> I'm very interested in any suggestions folks have to make that number
>>> smaller.  I know Chromium runs the layout tests in parallel using their
>>> python run_webkit_tests.py harness.  Try-bots would be another solution to
>>> the "make sure it passes all the tests before landing".
>>>
>>> Again, very interested in suggestions as to how to improve this.
>>>
>>
>> We're working on try-bots for the ports of particular interest to Apple.
>> But independent of that I think speeding up the layout tests is a worthwhile
>> goal.
>>
>> Besides parallelism for better performance on multi-core systems, here are
>> some other ideas:
>>
>> - Look at the slowest tests and see if they can be made to run faster (I'm
>> running with --slowest as we speak).
>> - Look at the subdirectories that take the most time - it's possible that
>> in some cases we can consolidate multiple tests to make the whole test suite
>> run faster, if sheer number of tests is the bottleneck.
>>
>> I think a good goal to shoot for would be for the full test suite to run
>> in under 5 minutes on a reasonably modern laptop. That should allow for
>> reasonable use during development with some room for growth.
>>
>> Regards,
>> Maciej
>>
>>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090801/12f1455a/attachment.html>