[webkit-dev] Coming Soon: new-run-webkit-tests

[Yuzo Fujishima]

Hi, Eric,

In my development environment, new-run-webkit-tests reports 149 errors
(out of 12692 tests) while run-webkit-tests none. Is this expected?

Are all of the errors what you mean by "Exposes more flaky tests due to
running tests in a non-deterministic order" ?

Yuzo

On Sat, Apr 10, 2010 at 7:35 AM, Eric Seidel <eric at webkit.org> wrote:

> Thanks to the tireless efforts of Dirk Pranke, Ojan Vafai, and
> numerous other engineers, WebKit now has a new test harness for
> running the Layout Tests.

> It's FAST:
> 4-core Mac Pro, Debug build:
> run-webkit-tests: 11m25s
> new-run-webkit-tests: 3m24s
> new-run-webkit-tests --experimental-fully-parallel: 2m42 [1]

> WebKitTools/Scripts/new-run-webkit-tests is ready for you to play with
> today, however it is not ready to replace run-webkit-tests as our main
> testing script yet.

> Pros:
> - Fast (see above)
> - Skipping is no longer our only tool, can mark tests as flaky (see
> test_expectations.txt).
> - Automatically retries failing tests to avoid failing due to flakiness.
> - Outputs json for easier integration with other tools (like flakiness
> dashboard).

> Cons:
> - Does not run on all core platforms yet (like Windows).
> - results.html output is not as pretty as run-webkit-tests' output.
> - Missing useful flags (e.g. --guard, --leaks).
> - Can't run webarchive, mathml or wml tests yet.
> - Exposes more flaky tests due to running tests in a non-deterministic
> order.

> The test_expectations.txt file format is currently documented in the
> file itself:

> http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac/test_expectations.txt

> I would invite you to try running new-run-webkit-tests on your Mac
> today.  You should feel encouraged to file bugs and hunt me down at
> next week's contributer meeting.

> If you're interested in contributing to the code, it can all be found  
> here:

> http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/webkitpy/layout_tests/

> Eventually we will rename run-webkit-tests to old-run-webkit-tests and
> new-run-webkit-tests to run-webkit-tests.  Since new-run-webkit-tests
> is a complete re-write, some features of "run-webkit-tests" may go
> missing when we move.  We will be moving them over to
> new-run-webkit-tests in priority order.

> Thanks for your time.

> -eric

> [1] new-run-webkit-tests currently defaults to a limited
> parallel-by-directory mode.  Once we get new-run-webkit-tests on by
> default and weed out the existing flaky tests we'll turn on the fully
> parallel-by-file mode for a greater speed boost.
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20100410/23a3ebf8/attachment.html>