Seems chromium tests are even slower: 10k * mean = 1,634s 50% (5000) of the tests are < 0.016s, assuming they all take that long, that's at most 80seconds. 40% (4000) of the tests are between 0.016 and 0.077, that's at most 311 seconds. 1,634 (total) - 80 - 311 leaves at least 1243 seconds accounted by the last 10% of tests! Assuming you and I both did our math correctly, the bulk of our delay is from a few slow tests, specifically less than 10% of the tests are accounting for 1243s (76%) of the time! -eric On Sat, Aug 1, 2009 at 12:52 PM, Ojan Vafai <ojan@chromium.org> wrote:
SUMMARY I doubt there's much room for improvement in the test harness itself outside of running tests in parallel. However, there's clearly a lot to be gained by making individual tests faster. For Chromium runs, the median time to run a test is ~16ms. It's not apples to apples, but I would be surprised if there were a significant difference between this and WebKit's run-webkit-tests.
With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the whole test suite.
DETAILS For the Chromium run_webkit_test.py, here's the stats in seconds from a TestShell run on Windows Debug. FWIW, we spit these stats out for every run. The eventual goal was to track this in a set of graphs, but I never got around to implementing that.
Median: 0.0160000324249 Mean: 0.163495878646 90th percentile: 0.077999830246 99th percentile: 3.1099998951 Standard deviation: 0.519017629449
Here's the times for the five slowest top-level directories: LayoutTests\editing took 43.5 seconds to run 976 tests. LayoutTests\svg took 53.7 seconds to run 891 tests. LayoutTests\fast took 378.7 seconds to run 3695 tests. LayoutTests\http took 396.9 seconds to run 632 tests. LayoutTests\media took 596.3 seconds to run 84 tests.
This might be Chromium specific, but I'm seeing many of the the media layout tests taking 8-10 seconds.
Ojan
On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <eric@webkit.org> wrote:
Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests:
9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :)
I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.)
-eric
On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com>wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev