[webkit-dev] An update on new-run-webkit-tests

[Dirk Pranke]

Hi all,

I am getting increasingly close to having new-run-webkit-tests running
correctly on the apple mac platform with full feature parity to
old-run-webkit-tests. (And, of course, it continues to run on all of
the Chromium bots as well). I am hoping to close the remaining issues
blocking full parity over the next couple weeks.

You can track the issues where NRWT still falls behind ORWT here:

https://bugs.webkit.org/show_bug.cgi?id=34984

Note that I expect some tests to fail or be flaky under NRWT when it
is running multiple processes/threads at a time. Doing so can both
expose test dependencies on the environment or change the order of
tests that are run by a DumpRenderTree, and I don't consider those to
be bugs that should stop people from using NRWT. The new tool has a
"test_expectations" mechanism that can be used to track expected
failures and we should use it (IMO).

That said, if there are significant issues making the tool unstable,
causing lots of tests to fail, or just missing functionality, now's
the time to let me know.

Also, for anyone building / maintaining webkit ports other than the
chromium and apple ones, I strongly encourage you to take another look
at getting NRWT running on your implementations. I hope to at least
make some attempt to run some of the GTK and QT bots myself, but I'm
not about to sign up to test every build variant personally :)

Note that  I believe that NRWT is fully usable today and people should
seriously start using it. That said, I believe the following bugs
should be fixed before we attempt to switch the apple mac port over.
You may wish to wait for these fixes before switching your port over
as well:

56731 new-run-webkit-tests doesn't support sample-on-timeout
56729 new-run-webkit-tests doesn't support WebKit2
55907 new-run-webkit-tests should upload crash logs
37739 new-run-webkit-tests does not report stderr output
37736 new-run-webkit-tests output is different from (and worse than)
original run-webkit-tests

There are also a number of issues keeping the apple win port from
working at the moment that I plan to work on shortly.

There are also a number of bugs currently listed as blocking that I
don't think really qualify. Unless told otherwise, I'm plannning to
remove the blocking flag from the following on Monday 4/11 (if they
haven't been fixed first):

57640 [GTK] overlapping drag&drop tests fail on NRWT
55909 new-run-webkit-tests --run-singly option is busted
55163 new-run-webkit-tests: enable multiple processes by default on Chromium Win
47240 new-run-webkit-tests: getting an "error 2" back from ImageDiff
37426 new-run-webkit-tests should use the ServerProcess abstraction in
chromium.py
37007 fast/tokenizer/doctype-search-reset.html fails when run out of
order (new-run-webkit-tests)
35359 fast/repaint/renderer-destruction-by-invalidateSelection-crash.html
fails intermittently
35266 new-run-webkit-tests --platform=mac-leopard timeout limit should
match run-webkit-tests
35049 http/tests/security/cross-frame-access-put.html fails
intermittently under new-run-webkit-tests
35006 fast/dom/global-constructors.html is failing based on previous tests

Also, just because I don't think they should block a cutover, I do
still think they should be fixed, so don't worry about that :)

Lastly, over the next couple weeks I also plan to produce some better
documentation for both how to use NRWT and how the code itself is
structured for anyone that wants to hack on it or port it to new
platforms.

Comments definitely welcome,

-- Dirk