[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

[Dirk Pranke]

On Thu, Jul 7, 2011 at 10:27 AM, Tony Chang <tony at chromium.org> wrote:
> One difference with the chromium port is that we try to use a single
> test_expectations.txt that covers all platforms and OS versions (win xp,
> vista, 7, mac leopard, snow leopard, linux 32, 64, GPU vs CPU, Debug vs
> Release).  The tokens to the left of the test name specify what
> configuration the expectation applies to.  Because of that, there hasn't
> been much need for multiple test_expectations.txt files.
> There is some code already in NRWT for cascading test_expectations.txt.
>  Currently, it's specific to the chromium port where we merge
> the test_expectations.txt in the webkit repo with a test_expectations.txt
> file in the chromium repo (it just concatenates them together).  It would be
> pretty straight forward to make this code generic for all ports.
> It seems like we have a few options.  We could have a separate
> test_expectations.txt per layout test platform directory and have cascade
> logic hard coded into NRWT or with an #include directive.  At the other
> extreme, we could have a single monolithic test_expectations.txt file that
> knows about all platforms.  Or something in the middle: have a
> test_expectations.txt for mac, mac-leopard, mac-snowleopard, one for qt*,
> one for all the WebKit2 ports, etc.  I suspect we'll want to go with
> something in the middle.

Tony's description is spot-on. The only reason we don't support
cascading expectations files is because it wasn't clear to me how we
would want things to work (i.e., which of the choices above) and I
wasn't able to get much input a few months ago.

If there is a consensus, it will be easy to implement, so how do we
actually want this to work?

-- Dirk

> On Thu, Jul 7, 2011 at 10:06 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>> On Jul 7, 2011, at 10:03 AM, Eric Seidel wrote:
>>
>> > I do not know the history as to why Chromium removed support for
>> > test_expectations cascading.
>> >
>> > Ideally we would have fewer test expectations, not more in the future.
>> > :)
>>
>> The cascading is really really useful for supporting multiple different
>> Mac OS X versions with different results, and WebKit2 as an orthogonal
>> dimension. Perhaps one possibility is to have something like an include
>> directive in the expectations file, so the cascading can be defined by the
>> expectations files themselves, rather than hardcoded in scripts.
>>
>> Regards,
>> Maciej
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>