[webkit-dev] Coming Soon: new-run-webkit-tests

[Eric Seidel]

Thanks to the tireless efforts of Dirk Pranke, Ojan Vafai, and
numerous other engineers, WebKit now has a new test harness for
running the Layout Tests.

It's FAST:
4-core Mac Pro, Debug build:
run-webkit-tests: 11m25s
new-run-webkit-tests: 3m24s
new-run-webkit-tests --experimental-fully-parallel: 2m42 [1]

WebKitTools/Scripts/new-run-webkit-tests is ready for you to play with
today, however it is not ready to replace run-webkit-tests as our main
testing script yet.

Pros:
- Fast (see above)
- Skipping is no longer our only tool, can mark tests as flaky (see
test_expectations.txt).
- Automatically retries failing tests to avoid failing due to flakiness.
- Outputs json for easier integration with other tools (like flakiness
dashboard).

Cons:
- Does not run on all core platforms yet (like Windows).
- results.html output is not as pretty as run-webkit-tests' output.
- Missing useful flags (e.g. --guard, --leaks).
- Can't run webarchive, mathml or wml tests yet.
- Exposes more flaky tests due to running tests in a non-deterministic order.

The test_expectations.txt file format is currently documented in the
file itself:
http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac/test_expectations.txt

I would invite you to try running new-run-webkit-tests on your Mac
today.  You should feel encouraged to file bugs and hunt me down at
next week's contributer meeting.

If you're interested in contributing to the code, it can all be found here:
http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/webkitpy/layout_tests/

Eventually we will rename run-webkit-tests to old-run-webkit-tests and
new-run-webkit-tests to run-webkit-tests.  Since new-run-webkit-tests
is a complete re-write, some features of "run-webkit-tests" may go
missing when we move.  We will be moving them over to
new-run-webkit-tests in priority order.

Thanks for your time.

-eric

[1] new-run-webkit-tests currently defaults to a limited
parallel-by-directory mode.  Once we get new-run-webkit-tests on by
default and weed out the existing flaky tests we'll turn on the fully
parallel-by-file mode for a greater speed boost.