[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

Eric Seidel eric at webkit.org
Wed Jul 6 09:29:15 PDT 2011

One other difference I should point out:

Skipped lists "cascade" through the normal results fallback mechanisms
(if it ever doesn't exactly match ORWT, that's a bug!).  For example,
Mac uses platform/mac-snowleopard/Skipped, combined with

test_expectations do not.  test_expectations only use your "base port
name" (ignoring the version).  For example,

NRWT uses both!  It will read in all the port's Skipped files, covert
them to SKIP text_expectations, and add them to your test_expectations

For better or worse, NRWT will error out, if you have duplicates in
your test_expectations file, including duplicates between your
test_expectations file and your Skipped lists.

Hopefully this makes sense.  We can of course change all this behavior
if folks need in the future.


On Wed, Jul 6, 2011 at 8:58 AM, Adam Barth <abarth at webkit.org> wrote:
> I'm not sure we've quite figured that out yet.  NRWT supports both
> Skipped lists and test_expectations.txt, which is a more expressive
> (but also more complex) version of Skipped lists.  IMHO, we should
> wait for the dust to settle on the transition before changing our
> practices.
> Adam
> On Wed, Jul 6, 2011 at 8:53 AM, Adam Roben <aroben at apple.com> wrote:
>> Now that more and more ports are switching to NRWT, it would be great for someone to explain what the best practices are for dealing with failing and flaky tests.
>> -Adam
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

More information about the webkit-dev mailing list