[webkit-dev] LayoutTests take too long to run

[Eric Seidel]

Yeah, I stared a slowest run before going to bed, and got the following:
The 10 slowest tests:

9.58 secs: editing/selection/move-left-right.html
9.37 secs: fast/js/array-filter.html
7.71 secs: editing/selection/extend-selection.html
6.82 secs: fast/js/sort-randomly.html
6.25 secs: http/tests/cache/subresource-expiration.html
5.77 secs: fast/js/array-enumerators-functions.html
5.41 secs: fast/events/click-count.html
5.39 secs: fast/js/toString-and-valueOf-override.html
5.26 secs: fast/js/try-catch-crash.html
5.05 secs: http/tests/navigation/slowmetaredirect-basic.html

that's 10% of our time right there. :)  I bet some of those have 5 second
timers and should be easy to fix.  I'll take a look, anyone else who would
like to help would be most welcome. :)

I remember Ojan finding that running tests in parallel was the biggest
overall win, but there is definitely a lot of low-hanging fruit in the
slowest tests too.  I'm curious what the fastest tests run in.  I would
expect the harness cost to be near-zero, but I'm not sure.  (i.e. what's the
runtime of run-webkit-tests with 10000 empty test.  I would expect only a
few seconds.)

-eric

On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
>
>  681.70s total testing time
>>
>> That's 11.5 minutes for every patch I want to land.  (because I run the
>> layout tests before landing anything, as part of bugzilla-tool).
>>
>> I'm very interested in any suggestions folks have to make that number
>> smaller.  I know Chromium runs the layout tests in parallel using their
>> python run_webkit_tests.py harness.  Try-bots would be another solution to
>> the "make sure it passes all the tests before landing".
>>
>> Again, very interested in suggestions as to how to improve this.
>>
>
> We're working on try-bots for the ports of particular interest to Apple.
> But independent of that I think speeding up the layout tests is a worthwhile
> goal.
>
> Besides parallelism for better performance on multi-core systems, here are
> some other ideas:
>
> - Look at the slowest tests and see if they can be made to run faster (I'm
> running with --slowest as we speak).
> - Look at the subdirectories that take the most time - it's possible that
> in some cases we can consolidate multiple tests to make the whole test suite
> run faster, if sheer number of tests is the bottleneck.
>
> I think a good goal to shoot for would be for the full test suite to run in
> under 5 minutes on a reasonably modern laptop. That should allow for
> reasonable use during development with some room for growth.
>
> Regards,
> Maciej
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090801/40e525bd/attachment.html>