[webkit-dev] An update on new-run-webkit-tests

[Dirk Pranke]

On Wed, Apr 6, 2011 at 7:39 PM, Dirk Pranke <dpranke at chromium.org> wrote:
> Hi all,
>
> I am getting increasingly close to having new-run-webkit-tests running
> correctly on the apple mac platform with full feature parity to
> old-run-webkit-tests. (And, of course, it continues to run on all of
> the Chromium bots as well). I am hoping to close the remaining issues
> blocking full parity over the next couple weeks.
>
> You can track the issues where NRWT still falls behind ORWT here:
>
> https://bugs.webkit.org/show_bug.cgi?id=34984
>
> Note that I expect some tests to fail or be flaky under NRWT when it
> is running multiple processes/threads at a time. Doing so can both
> expose test dependencies on the environment or change the order of
> tests that are run by a DumpRenderTree, and I don't consider those to
> be bugs that should stop people from using NRWT. The new tool has a
> "test_expectations" mechanism that can be used to track expected
> failures and we should use it (IMO).
>
> That said, if there are significant issues making the tool unstable,
> causing lots of tests to fail, or just missing functionality, now's
> the time to let me know.
>
> Also, for anyone building / maintaining webkit ports other than the
> chromium and apple ones, I strongly encourage you to take another look
> at getting NRWT running on your implementations. I hope to at least
> make some attempt to run some of the GTK and QT bots myself, but I'm
> not about to sign up to test every build variant personally :)
>
> Note that  I believe that NRWT is fully usable today and people should
> seriously start using it. That said, I believe the following bugs
> should be fixed before we attempt to switch the apple mac port over.
> You may wish to wait for these fixes before switching your port over
> as well:
>
> 56731 new-run-webkit-tests doesn't support sample-on-timeout
> 56729 new-run-webkit-tests doesn't support WebKit2
> 55907 new-run-webkit-tests should upload crash logs
> 37739 new-run-webkit-tests does not report stderr output
> 37736 new-run-webkit-tests output is different from (and worse than)
> original run-webkit-tests
>
> There are also a number of issues keeping the apple win port from
> working at the moment that I plan to work on shortly.
>
> There are also a number of bugs currently listed as blocking that I
> don't think really qualify. Unless told otherwise, I'm plannning to
> remove the blocking flag from the following on Monday 4/11 (if they
> haven't been fixed first):
>
> 57640 [GTK] overlapping drag&drop tests fail on NRWT

Minor revision ... this one might actually qualify as a blocker :)

-- Dirk

> 55909 new-run-webkit-tests --run-singly option is busted
> 55163 new-run-webkit-tests: enable multiple processes by default on Chromium Win
> 47240 new-run-webkit-tests: getting an "error 2" back from ImageDiff
> 37426 new-run-webkit-tests should use the ServerProcess abstraction in
> chromium.py
> 37007 fast/tokenizer/doctype-search-reset.html fails when run out of
> order (new-run-webkit-tests)
> 35359 fast/repaint/renderer-destruction-by-invalidateSelection-crash.html
> fails intermittently
> 35266 new-run-webkit-tests --platform=mac-leopard timeout limit should
> match run-webkit-tests
> 35049 http/tests/security/cross-frame-access-put.html fails
> intermittently under new-run-webkit-tests
> 35006 fast/dom/global-constructors.html is failing based on previous tests
>
> Also, just because I don't think they should block a cutover, I do
> still think they should be fixed, so don't worry about that :)
>
> Lastly, over the next couple weeks I also plan to produce some better
> documentation for both how to use NRWT and how the code itself is
> structured for anyone that wants to hack on it or port it to new
> platforms.
>
> Comments definitely welcome,
>
> -- Dirk
>