Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests: 9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :) I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.) -eric On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com> wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej