[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

[Eric Seidel]

One other difference I should point out:

Skipped lists "cascade" through the normal results fallback mechanisms
(if it ever doesn't exactly match ORWT, that's a bug!).  For example,
Mac uses platform/mac-snowleopard/Skipped, combined with
platform/mac/Skipped.
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/webkit.py#L288

test_expectations do not.  test_expectations only use your "base port
name" (ignoring the version).  For example,
platform/mac/test_expectations.txt.
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/webkit.py#L73

NRWT uses both!  It will read in all the port's Skipped files, covert
them to SKIP text_expectations, and add them to your test_expectations
file.
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/webkit.py#L309

For better or worse, NRWT will error out, if you have duplicates in
your test_expectations file, including duplicates between your
test_expectations file and your Skipped lists.

Hopefully this makes sense.  We can of course change all this behavior
if folks need in the future.

-eric

On Wed, Jul 6, 2011 at 8:58 AM, Adam Barth <abarth at webkit.org> wrote:
> I'm not sure we've quite figured that out yet.  NRWT supports both
> Skipped lists and test_expectations.txt, which is a more expressive
> (but also more complex) version of Skipped lists.  IMHO, we should
> wait for the dust to settle on the transition before changing our
> practices.
>
> Adam
>
>
> On Wed, Jul 6, 2011 at 8:53 AM, Adam Roben <aroben at apple.com> wrote:
>> Now that more and more ports are switching to NRWT, it would be great for someone to explain what the best practices are for dealing with failing and flaky tests.
>>
>> -Adam
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>