[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

[Tony Chang]

One difference with the chromium port is that we try to use a single
test_expectations.txt that covers all platforms and OS versions (win xp,
vista, 7, mac leopard, snow leopard, linux 32, 64, GPU vs CPU, Debug vs
Release).  The tokens to the left of the test name specify what
configuration the expectation applies to.  Because of that, there hasn't
been much need for multiple test_expectations.txt files.

There is some code already in NRWT for cascading test_expectations.txt.
 Currently, it's specific to the chromium port where we merge
the test_expectations.txt in the webkit repo with a test_expectations.txt
file in the chromium repo (it just concatenates them together).  It would be
pretty straight forward to make this code generic for all ports.

It seems like we have a few options.  We could have a separate
test_expectations.txt per layout test platform directory and have cascade
logic hard coded into NRWT or with an #include directive.  At the other
extreme, we could have a single monolithic test_expectations.txt file that
knows about all platforms.  Or something in the middle: have a
test_expectations.txt for mac, mac-leopard, mac-snowleopard, one for qt*,
one for all the WebKit2 ports, etc.  I suspect we'll want to go with
something in the middle.

On Thu, Jul 7, 2011 at 10:06 AM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Jul 7, 2011, at 10:03 AM, Eric Seidel wrote:
>
> > I do not know the history as to why Chromium removed support for
> > test_expectations cascading.
> >
> > Ideally we would have fewer test expectations, not more in the future. :)
>
> The cascading is really really useful for supporting multiple different Mac
> OS X versions with different results, and WebKit2 as an orthogonal
> dimension. Perhaps one possibility is to have something like an include
> directive in the expectations file, so the cascading can be defined by the
> expectations files themselves, rather than hardcoded in scripts.
>
> Regards,
> Maciej
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20110707/c214f52d/attachment.html>