[webkit-dev] LayoutTests take too long to run

[Ojan Vafai]

On Sat, Aug 1, 2009 at 4:23 PM, Eric Seidel <eric at webkit.org> wrote:

> Seems chromium tests are even slower: 10k * mean = 1,634s
>

For completeness sake, the Chromium numbers include the pixel tests.


> 50% (5000) of the tests are < 0.016s, assuming they all take that long,
> that's at most 80seconds.
> 40% (4000) of the tests are between 0.016 and 0.077, that's at most 311
> seconds.
>
> 1,634 (total) - 80 - 311 leaves at least 1243 seconds accounted by the last
> 10% of tests!
>
> Assuming you and I both did our math correctly, the bulk of our delay is
> from a few slow tests, specifically less
> than 10% of the tests are accounting for 1243s (76%) of the time!
>
> -eric
>
> On Sat, Aug 1, 2009 at 12:52 PM, Ojan Vafai <ojan at chromium.org> wrote:
>
>> SUMMARY
>> I doubt there's much room for improvement in the test harness itself
>> outside of running tests in parallel. However, there's clearly a lot to be
>> gained by making individual tests faster. For Chromium runs, the median time
>> to run a test is ~16ms. It's not apples to apples, but I would be surprised
>> if there were a significant difference between this and WebKit's
>> run-webkit-tests.
>>
>> With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the
>> whole test suite.
>>
>> DETAILS
>> For the Chromium run_webkit_test.py, here's the stats in seconds from a
>> TestShell run on Windows Debug. FWIW, we spit these stats out for every run.
>> The eventual goal was to track this in a set of graphs, but I never got
>> around to implementing that.
>>
>> Median: 0.0160000324249
>> Mean: 0.163495878646
>> 90th percentile: 0.077999830246
>> 99th percentile: 3.1099998951
>> Standard deviation: 0.519017629449
>>
>> Here's the times for the five slowest top-level directories:
>>  LayoutTests\editing took 43.5 seconds to run 976 tests.
>> LayoutTests\svg took 53.7 seconds to run 891 tests.
>> LayoutTests\fast took 378.7 seconds to run 3695 tests.
>> LayoutTests\http took 396.9 seconds to run 632 tests.
>> LayoutTests\media took 596.3 seconds to run 84 tests.
>>
>> This might be Chromium specific, but I'm seeing many of the the media
>> layout tests taking 8-10 seconds.
>>
>> Ojan
>>
>> On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <eric at webkit.org> wrote:
>>
>>> Yeah, I stared a slowest run before going to bed, and got the following:
>>> The 10 slowest tests:
>>>
>>> 9.58 secs: editing/selection/move-left-right.html
>>> 9.37 secs: fast/js/array-filter.html
>>> 7.71 secs: editing/selection/extend-selection.html
>>> 6.82 secs: fast/js/sort-randomly.html
>>> 6.25 secs: http/tests/cache/subresource-expiration.html
>>> 5.77 secs: fast/js/array-enumerators-functions.html
>>> 5.41 secs: fast/events/click-count.html
>>> 5.39 secs: fast/js/toString-and-valueOf-override.html
>>> 5.26 secs: fast/js/try-catch-crash.html
>>> 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
>>>
>>> that's 10% of our time right there. :)  I bet some of those have 5 second
>>> timers and should be easy to fix.  I'll take a look, anyone else who would
>>> like to help would be most welcome. :)
>>>
>>> I remember Ojan finding that running tests in parallel was the biggest
>>> overall win, but there is definitely a lot of low-hanging fruit in the
>>> slowest tests too.  I'm curious what the fastest tests run in.  I would
>>> expect the harness cost to be near-zero, but I'm not sure.  (i.e. what's the
>>> runtime of run-webkit-tests with 10000 empty test.  I would expect only a
>>> few seconds.)
>>>
>>> -eric
>>>
>>> On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs at apple.com>wrote:
>>>
>>>>
>>>> On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
>>>>
>>>>  681.70s total testing time
>>>>>
>>>>> That's 11.5 minutes for every patch I want to land.  (because I run the
>>>>> layout tests before landing anything, as part of bugzilla-tool).
>>>>>
>>>>> I'm very interested in any suggestions folks have to make that number
>>>>> smaller.  I know Chromium runs the layout tests in parallel using their
>>>>> python run_webkit_tests.py harness.  Try-bots would be another solution to
>>>>> the "make sure it passes all the tests before landing".
>>>>>
>>>>> Again, very interested in suggestions as to how to improve this.
>>>>>
>>>>
>>>> We're working on try-bots for the ports of particular interest to Apple.
>>>> But independent of that I think speeding up the layout tests is a worthwhile
>>>> goal.
>>>>
>>>> Besides parallelism for better performance on multi-core systems, here
>>>> are some other ideas:
>>>>
>>>> - Look at the slowest tests and see if they can be made to run faster
>>>> (I'm running with --slowest as we speak).
>>>> - Look at the subdirectories that take the most time - it's possible
>>>> that in some cases we can consolidate multiple tests to make the whole test
>>>> suite run faster, if sheer number of tests is the bottleneck.
>>>>
>>>> I think a good goal to shoot for would be for the full test suite to run
>>>> in under 5 minutes on a reasonably modern laptop. That should allow for
>>>> reasonable use during development with some room for growth.
>>>>
>>>> Regards,
>>>> Maciej
>>>>
>>>>
>>>
>>> _______________________________________________
>>> webkit-dev mailing list
>>> webkit-dev at lists.webkit.org
>>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090801/0a493e57/attachment.html>