One other difference I should point out: Skipped lists "cascade" through the normal results fallback mechanisms (if it ever doesn't exactly match ORWT, that's a bug!). For example, Mac uses platform/mac-snowleopard/Skipped, combined with platform/mac/Skipped. http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/por... test_expectations do not. test_expectations only use your "base port name" (ignoring the version). For example, platform/mac/test_expectations.txt. http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/por... NRWT uses both! It will read in all the port's Skipped files, covert them to SKIP text_expectations, and add them to your test_expectations file. http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/por... For better or worse, NRWT will error out, if you have duplicates in your test_expectations file, including duplicates between your test_expectations file and your Skipped lists. Hopefully this makes sense. We can of course change all this behavior if folks need in the future. -eric On Wed, Jul 6, 2011 at 8:58 AM, Adam Barth <abarth@webkit.org> wrote:
I'm not sure we've quite figured that out yet. NRWT supports both Skipped lists and test_expectations.txt, which is a more expressive (but also more complex) version of Skipped lists. IMHO, we should wait for the dust to settle on the transition before changing our practices.
Adam
On Wed, Jul 6, 2011 at 8:53 AM, Adam Roben <aroben@apple.com> wrote:
Now that more and more ports are switching to NRWT, it would be great for someone to explain what the best practices are for dealing with failing and flaky tests.
-Adam
webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev