[webkit-dev] LayoutTests take too long to run

[Eric Seidel]

Seems chromium tests are even slower: 10k * mean = 1,634s

50% (5000) of the tests are < 0.016s, assuming they all take that long,
that's at most 80seconds.
40% (4000) of the tests are between 0.016 and 0.077, that's at most 311
seconds.

1,634 (total) - 80 - 311 leaves at least 1243 seconds accounted by the last
10% of tests!

Assuming you and I both did our math correctly, the bulk of our delay is
from a few slow tests, specifically less
than 10% of the tests are accounting for 1243s (76%) of the time!

-eric

On Sat, Aug 1, 2009 at 12:52 PM, Ojan Vafai <ojan at chromium.org> wrote:

> SUMMARY
> I doubt there's much room for improvement in the test harness itself
> outside of running tests in parallel. However, there's clearly a lot to be
> gained by making individual tests faster. For Chromium runs, the median time
> to run a test is ~16ms. It's not apples to apples, but I would be surprised
> if there were a significant difference between this and WebKit's
> run-webkit-tests.
>
> With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the
> whole test suite.
>
> DETAILS
> For the Chromium run_webkit_test.py, here's the stats in seconds from a
> TestShell run on Windows Debug. FWIW, we spit these stats out for every run.
> The eventual goal was to track this in a set of graphs, but I never got
> around to implementing that.
>
> Median: 0.0160000324249
> Mean: 0.163495878646
> 90th percentile: 0.077999830246
> 99th percentile: 3.1099998951
> Standard deviation: 0.519017629449
>
> Here's the times for the five slowest top-level directories:
>  LayoutTests\editing took 43.5 seconds to run 976 tests.
> LayoutTests\svg took 53.7 seconds to run 891 tests.
> LayoutTests\fast took 378.7 seconds to run 3695 tests.
> LayoutTests\http took 396.9 seconds to run 632 tests.
> LayoutTests\media took 596.3 seconds to run 84 tests.
>
> This might be Chromium specific, but I'm seeing many of the the media
> layout tests taking 8-10 seconds.
>
> Ojan
>
> On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <eric at webkit.org> wrote:
>
>> Yeah, I stared a slowest run before going to bed, and got the following:
>> The 10 slowest tests:
>>
>> 9.58 secs: editing/selection/move-left-right.html
>> 9.37 secs: fast/js/array-filter.html
>> 7.71 secs: editing/selection/extend-selection.html
>> 6.82 secs: fast/js/sort-randomly.html
>> 6.25 secs: http/tests/cache/subresource-expiration.html
>> 5.77 secs: fast/js/array-enumerators-functions.html
>> 5.41 secs: fast/events/click-count.html
>> 5.39 secs: fast/js/toString-and-valueOf-override.html
>> 5.26 secs: fast/js/try-catch-crash.html
>> 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
>>
>> that's 10% of our time right there. :)  I bet some of those have 5 second
>> timers and should be easy to fix.  I'll take a look, anyone else who would
>> like to help would be most welcome. :)
>>
>> I remember Ojan finding that running tests in parallel was the biggest
>> overall win, but there is definitely a lot of low-hanging fruit in the
>> slowest tests too.  I'm curious what the fastest tests run in.  I would
>> expect the harness cost to be near-zero, but I'm not sure.  (i.e. what's the
>> runtime of run-webkit-tests with 10000 empty test.  I would expect only a
>> few seconds.)
>>
>> -eric
>>
>> On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs at apple.com>wrote:
>>
>>>
>>> On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
>>>
>>>  681.70s total testing time
>>>>
>>>> That's 11.5 minutes for every patch I want to land.  (because I run the
>>>> layout tests before landing anything, as part of bugzilla-tool).
>>>>
>>>> I'm very interested in any suggestions folks have to make that number
>>>> smaller.  I know Chromium runs the layout tests in parallel using their
>>>> python run_webkit_tests.py harness.  Try-bots would be another solution to
>>>> the "make sure it passes all the tests before landing".
>>>>
>>>> Again, very interested in suggestions as to how to improve this.
>>>>
>>>
>>> We're working on try-bots for the ports of particular interest to Apple.
>>> But independent of that I think speeding up the layout tests is a worthwhile
>>> goal.
>>>
>>> Besides parallelism for better performance on multi-core systems, here
>>> are some other ideas:
>>>
>>> - Look at the slowest tests and see if they can be made to run faster
>>> (I'm running with --slowest as we speak).
>>> - Look at the subdirectories that take the most time - it's possible that
>>> in some cases we can consolidate multiple tests to make the whole test suite
>>> run faster, if sheer number of tests is the bottleneck.
>>>
>>> I think a good goal to shoot for would be for the full test suite to run
>>> in under 5 minutes on a reasonably modern laptop. That should allow for
>>> reasonable use during development with some room for growth.
>>>
>>> Regards,
>>> Maciej
>>>
>>>
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090801/1a956b83/attachment.html>