LayoutTests take too long to run
681.70s total testing time That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool). I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing". Again, very interested in suggestions as to how to improve this. -eric
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal. Besides parallelism for better performance on multi-core systems, here are some other ideas: - Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck. I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth. Regards, Maciej
Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests: 9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :) I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.) -eric On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com> wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej
SUMMARY I doubt there's much room for improvement in the test harness itself outside of running tests in parallel. However, there's clearly a lot to be gained by making individual tests faster. For Chromium runs, the median time to run a test is ~16ms. It's not apples to apples, but I would be surprised if there were a significant difference between this and WebKit's run-webkit-tests. With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the whole test suite. DETAILS For the Chromium run_webkit_test.py, here's the stats in seconds from a TestShell run on Windows Debug. FWIW, we spit these stats out for every run. The eventual goal was to track this in a set of graphs, but I never got around to implementing that. Median: 0.0160000324249 Mean: 0.163495878646 90th percentile: 0.077999830246 99th percentile: 3.1099998951 Standard deviation: 0.519017629449 Here's the times for the five slowest top-level directories: LayoutTests\editing took 43.5 seconds to run 976 tests. LayoutTests\svg took 53.7 seconds to run 891 tests. LayoutTests\fast took 378.7 seconds to run 3695 tests. LayoutTests\http took 396.9 seconds to run 632 tests. LayoutTests\media took 596.3 seconds to run 84 tests. This might be Chromium specific, but I'm seeing many of the the media layout tests taking 8-10 seconds. Ojan On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <eric@webkit.org> wrote:
Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests:
9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :)
I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.)
-eric
On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com> wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
On Aug 1, 2009, at 10:52 AM, Ojan Vafai wrote:
Here's the times for the five slowest top-level directories: LayoutTests\editing took 43.5 seconds to run 976 tests. LayoutTests\svg took 53.7 seconds to run 891 tests. LayoutTests\fast took 378.7 seconds to run 3695 tests. LayoutTests\http took 396.9 seconds to run 632 tests. LayoutTests\media took 596.3 seconds to run 84 tests.
This might be Chromium specific, but I'm seeing many of the the media layout tests taking 8-10 seconds.
That seems unusually high, on a MacBook with TOT debug build the slowest test (video-played.html) takes less than 4 seconds. Running all of the media tests takes ~40 seconds: eric% run-webkit-tests --debug --slowest LayoutTests/media/ 40.87s total testing time The 10 slowest tests: 3.23 secs: media/video-played.html 2.53 secs: media/audio-constructor-autobuffer.html 2.05 secs: media/video-played-collapse.html 1.93 secs: media/video-loop.html 1.53 secs: media/controls-after-reload.html 1.39 secs: media/event-attributes.html 1.35 secs: media/video-aspect-ratio.html 1.16 secs: media/video-timeupdate-during-playback.html 1.12 secs: media/video-played-reset.html 1.09 secs: media/audio-data-url.html eric
Seems chromium tests are even slower: 10k * mean = 1,634s 50% (5000) of the tests are < 0.016s, assuming they all take that long, that's at most 80seconds. 40% (4000) of the tests are between 0.016 and 0.077, that's at most 311 seconds. 1,634 (total) - 80 - 311 leaves at least 1243 seconds accounted by the last 10% of tests! Assuming you and I both did our math correctly, the bulk of our delay is from a few slow tests, specifically less than 10% of the tests are accounting for 1243s (76%) of the time! -eric On Sat, Aug 1, 2009 at 12:52 PM, Ojan Vafai <ojan@chromium.org> wrote:
SUMMARY I doubt there's much room for improvement in the test harness itself outside of running tests in parallel. However, there's clearly a lot to be gained by making individual tests faster. For Chromium runs, the median time to run a test is ~16ms. It's not apples to apples, but I would be surprised if there were a significant difference between this and WebKit's run-webkit-tests.
With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the whole test suite.
DETAILS For the Chromium run_webkit_test.py, here's the stats in seconds from a TestShell run on Windows Debug. FWIW, we spit these stats out for every run. The eventual goal was to track this in a set of graphs, but I never got around to implementing that.
Median: 0.0160000324249 Mean: 0.163495878646 90th percentile: 0.077999830246 99th percentile: 3.1099998951 Standard deviation: 0.519017629449
Here's the times for the five slowest top-level directories: LayoutTests\editing took 43.5 seconds to run 976 tests. LayoutTests\svg took 53.7 seconds to run 891 tests. LayoutTests\fast took 378.7 seconds to run 3695 tests. LayoutTests\http took 396.9 seconds to run 632 tests. LayoutTests\media took 596.3 seconds to run 84 tests.
This might be Chromium specific, but I'm seeing many of the the media layout tests taking 8-10 seconds.
Ojan
On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <eric@webkit.org> wrote:
Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests:
9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :)
I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.)
-eric
On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com>wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
On Sat, Aug 1, 2009 at 4:23 PM, Eric Seidel <eric@webkit.org> wrote:
Seems chromium tests are even slower: 10k * mean = 1,634s
For completeness sake, the Chromium numbers include the pixel tests.
50% (5000) of the tests are < 0.016s, assuming they all take that long, that's at most 80seconds. 40% (4000) of the tests are between 0.016 and 0.077, that's at most 311 seconds.
1,634 (total) - 80 - 311 leaves at least 1243 seconds accounted by the last 10% of tests!
Assuming you and I both did our math correctly, the bulk of our delay is from a few slow tests, specifically less than 10% of the tests are accounting for 1243s (76%) of the time!
-eric
On Sat, Aug 1, 2009 at 12:52 PM, Ojan Vafai <ojan@chromium.org> wrote:
SUMMARY I doubt there's much room for improvement in the test harness itself outside of running tests in parallel. However, there's clearly a lot to be gained by making individual tests faster. For Chromium runs, the median time to run a test is ~16ms. It's not apples to apples, but I would be surprised if there were a significant difference between this and WebKit's run-webkit-tests.
With 16ms / test and 10,000 tests, it would take ~2.7 minutes to run the whole test suite.
DETAILS For the Chromium run_webkit_test.py, here's the stats in seconds from a TestShell run on Windows Debug. FWIW, we spit these stats out for every run. The eventual goal was to track this in a set of graphs, but I never got around to implementing that.
Median: 0.0160000324249 Mean: 0.163495878646 90th percentile: 0.077999830246 99th percentile: 3.1099998951 Standard deviation: 0.519017629449
Here's the times for the five slowest top-level directories: LayoutTests\editing took 43.5 seconds to run 976 tests. LayoutTests\svg took 53.7 seconds to run 891 tests. LayoutTests\fast took 378.7 seconds to run 3695 tests. LayoutTests\http took 396.9 seconds to run 632 tests. LayoutTests\media took 596.3 seconds to run 84 tests.
This might be Chromium specific, but I'm seeing many of the the media layout tests taking 8-10 seconds.
Ojan
On Sat, Aug 1, 2009 at 12:49 PM, Eric Seidel <eric@webkit.org> wrote:
Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests:
9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :)
I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.)
-eric
On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com>wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
For editing/selection/extend-selection.html, It seems like the bottleneck is testExtendingSelection, in particular, positionExtendingInDirection and checkSameOrder: function positionsExtendingInDirection(sel, direction, granularity) { var positions = []; while (true) { positions.push({ node: sel.extentNode, begin: sel.baseOffset, end: sel.extentOffset }); sel.modify("extend", direction, granularity); if (positions[positions.length - 1].node == sel.extentNode && positions[positions.length - 1].end == sel.extentOffset) break; }; return positions; } We return positions and compare the results for LTR and RTL using checkSameOrder function checkSameOrder(inputPositions, inputSamePositions) { var positions = inputPositions.slice(); var samePositions = inputSamePositions.slice(); var mismatch = positions.length != samePositions.length; if (mismatch) log("WARNING: positions should be the same, but the length are not the same " + positions.length + " vs. " + samePositions.length + "\n"); while (!mismatch) { var pos = positions.pop(); if (!pos) break; var samePos = samePositions.pop(); if (pos.node != samePos.node) { mismatch = true; log("WARNING: positions should be the same, but node are not the same\n"); } if (pos.begin != samePos.begin) { mismatch = true; log("WARNING: positions should be the same, but begin are not the same " + pos.begin + " vs. " + samePos.begin + "\n"); } if (pos.end != samePos.end) { mismatch = true; log("WARNING: positions should be the same, but end are not the same " + pos.end + " vs. " + samePos.end + "\n"); } } } Can anyone think of any way to make this faster? Ryosuke On Sat, Aug 1, 2009 at 9:49 AM, Eric Seidel <eric@webkit.org> wrote:
Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests:
9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :)
I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.)
-eric
On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com> wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
editing/selection/move-left-right.html has very similar script where we return positions and compare entires in reverse order. Since there is no setTimeout, we need to rewrite those scripts so that they run faster or improve the performance of WebKit. Ryosuke On Sat, Aug 1, 2009 at 1:09 PM, Ryosuke Niwa <rniwa@google.com> wrote:
For editing/selection/extend-selection.html, It seems like the bottleneck is testExtendingSelection, in particular, positionExtendingInDirection and checkSameOrder:
function positionsExtendingInDirection(sel, direction, granularity) { var positions = [];
while (true) { positions.push({ node: sel.extentNode, begin: sel.baseOffset, end: sel.extentOffset }); sel.modify("extend", direction, granularity); if (positions[positions.length - 1].node == sel.extentNode && positions[positions.length - 1].end == sel.extentOffset) break; }; return positions; }
We return positions and compare the results for LTR and RTL using checkSameOrder
function checkSameOrder(inputPositions, inputSamePositions) { var positions = inputPositions.slice(); var samePositions = inputSamePositions.slice(); var mismatch = positions.length != samePositions.length; if (mismatch) log("WARNING: positions should be the same, but the length are not the same " + positions.length + " vs. " + samePositions.length + "\n"); while (!mismatch) { var pos = positions.pop(); if (!pos) break; var samePos = samePositions.pop(); if (pos.node != samePos.node) { mismatch = true; log("WARNING: positions should be the same, but node are not the same\n"); } if (pos.begin != samePos.begin) { mismatch = true; log("WARNING: positions should be the same, but begin are not the same " + pos.begin + " vs. " + samePos.begin + "\n"); } if (pos.end != samePos.end) { mismatch = true; log("WARNING: positions should be the same, but end are not the same " + pos.end + " vs. " + samePos.end + "\n"); } } }
Can anyone think of any way to make this faster?
Ryosuke
On Sat, Aug 1, 2009 at 9:49 AM, Eric Seidel <eric@webkit.org> wrote:
Yeah, I stared a slowest run before going to bed, and got the following: The 10 slowest tests:
9.58 secs: editing/selection/move-left-right.html 9.37 secs: fast/js/array-filter.html 7.71 secs: editing/selection/extend-selection.html 6.82 secs: fast/js/sort-randomly.html 6.25 secs: http/tests/cache/subresource-expiration.html 5.77 secs: fast/js/array-enumerators-functions.html 5.41 secs: fast/events/click-count.html 5.39 secs: fast/js/toString-and-valueOf-override.html 5.26 secs: fast/js/try-catch-crash.html 5.05 secs: http/tests/navigation/slowmetaredirect-basic.html
that's 10% of our time right there. :) I bet some of those have 5 second timers and should be easy to fix. I'll take a look, anyone else who would like to help would be most welcome. :)
I remember Ojan finding that running tests in parallel was the biggest overall win, but there is definitely a lot of low-hanging fruit in the slowest tests too. I'm curious what the fastest tests run in. I would expect the harness cost to be near-zero, but I'm not sure. (i.e. what's the runtime of run-webkit-tests with 10000 empty test. I would expect only a few seconds.)
-eric
On Fri, Jul 31, 2009 at 11:21 PM, Maciej Stachowiak <mjs@apple.com>wrote:
On Jul 31, 2009, at 10:25 PM, Eric Seidel wrote:
681.70s total testing time
That's 11.5 minutes for every patch I want to land. (because I run the layout tests before landing anything, as part of bugzilla-tool).
I'm very interested in any suggestions folks have to make that number smaller. I know Chromium runs the layout tests in parallel using their python run_webkit_tests.py harness. Try-bots would be another solution to the "make sure it passes all the tests before landing".
Again, very interested in suggestions as to how to improve this.
We're working on try-bots for the ports of particular interest to Apple. But independent of that I think speeding up the layout tests is a worthwhile goal.
Besides parallelism for better performance on multi-core systems, here are some other ideas:
- Look at the slowest tests and see if they can be made to run faster (I'm running with --slowest as we speak). - Look at the subdirectories that take the most time - it's possible that in some cases we can consolidate multiple tests to make the whole test suite run faster, if sheer number of tests is the bottleneck.
I think a good goal to shoot for would be for the full test suite to run in under 5 minutes on a reasonably modern laptop. That should allow for reasonable use during development with some room for growth.
Regards, Maciej
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
participants (5)
-
Eric Carlson
-
Eric Seidel
-
Maciej Stachowiak
-
Ojan Vafai
-
Ryosuke Niwa