Hi, Eric,<div><br></div><div>In my development environment, new-run-webkit-tests reports 149 errors</div><div>(out of 12692 tests) while run-webkit-tests none. Is this expected?</div><div><br></div><div>Are all of the errors what you mean by "Exposes more flaky tests due to</div>
<div>running tests in a non-deterministic order" ?</div><meta charset="utf-8"><div><br></div><div>Yuzo<br><br><div class="gmail_quote">On Sat, Apr 10, 2010 at 7:35 AM, Eric Seidel <span dir="ltr"><<a href="mailto:eric@webkit.org">eric@webkit.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Thanks to the tireless efforts of Dirk Pranke, Ojan Vafai, and<br>
numerous other engineers, WebKit now has a new test harness for<br>
running the Layout Tests.<br>
<br>
It's FAST:<br>
4-core Mac Pro, Debug build:<br>
run-webkit-tests: 11m25s<br>
new-run-webkit-tests: 3m24s<br>
new-run-webkit-tests --experimental-fully-parallel: 2m42 [1]<br>
<br>
WebKitTools/Scripts/new-run-webkit-tests is ready for you to play with<br>
today, however it is not ready to replace run-webkit-tests as our main<br>
testing script yet.<br>
<br>
Pros:<br>
- Fast (see above)<br>
- Skipping is no longer our only tool, can mark tests as flaky (see<br>
test_expectations.txt).<br>
- Automatically retries failing tests to avoid failing due to flakiness.<br>
- Outputs json for easier integration with other tools (like flakiness<br>
dashboard).<br>
<br>
Cons:<br>
- Does not run on all core platforms yet (like Windows).<br>
- results.html output is not as pretty as run-webkit-tests' output.<br>
- Missing useful flags (e.g. --guard, --leaks).<br>
- Can't run webarchive, mathml or wml tests yet.<br>
- Exposes more flaky tests due to running tests in a non-deterministic order.<br>
<br>
The test_expectations.txt file format is currently documented in the<br>
file itself:<br>
<a href="http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac/test_expectations.txt" target="_blank">http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac/test_expectations.txt</a><br>
<br>
I would invite you to try running new-run-webkit-tests on your Mac<br>
today. You should feel encouraged to file bugs and hunt me down at<br>
next week's contributer meeting.<br>
<br>
If you're interested in contributing to the code, it can all be found here:<br>
<a href="http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/webkitpy/layout_tests/" target="_blank">http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/webkitpy/layout_tests/</a><br>
<br>
Eventually we will rename run-webkit-tests to old-run-webkit-tests and<br>
new-run-webkit-tests to run-webkit-tests. Since new-run-webkit-tests<br>
is a complete re-write, some features of "run-webkit-tests" may go<br>
missing when we move. We will be moving them over to<br>
new-run-webkit-tests in priority order.<br>
<br>
Thanks for your time.<br>
<br>
-eric<br>
<br>
[1] new-run-webkit-tests currently defaults to a limited<br>
parallel-by-directory mode. Once we get new-run-webkit-tests on by<br>
default and weed out the existing flaky tests we'll turn on the fully<br>
parallel-by-file mode for a greater speed boost.<br>
_______________________________________________<br>
webkit-dev mailing list<br>
<a href="mailto:webkit-dev@lists.webkit.org">webkit-dev@lists.webkit.org</a><br>
<a href="http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev" target="_blank">http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev</a><br>
</blockquote></div><br></div>