[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

[Maciej Stachowiak]

On Jul 7, 2011, at 10:39 AM, Dirk Pranke wrote:

> On Thu, Jul 7, 2011 at 10:27 AM, Tony Chang <tony at chromium.org> wrote:
>> One difference with the chromium port is that we try to use a single
>> test_expectations.txt that covers all platforms and OS versions (win xp,
>> vista, 7, mac leopard, snow leopard, linux 32, 64, GPU vs CPU, Debug vs
>> Release).  The tokens to the left of the test name specify what
>> configuration the expectation applies to.  Because of that, there hasn't
>> been much need for multiple test_expectations.txt files.
>> There is some code already in NRWT for cascading test_expectations.txt.
>>  Currently, it's specific to the chromium port where we merge
>> the test_expectations.txt in the webkit repo with a test_expectations.txt
>> file in the chromium repo (it just concatenates them together).  It would be
>> pretty straight forward to make this code generic for all ports.
>> It seems like we have a few options.  We could have a separate
>> test_expectations.txt per layout test platform directory and have cascade
>> logic hard coded into NRWT or with an #include directive.  At the other
>> extreme, we could have a single monolithic test_expectations.txt file that
>> knows about all platforms.  Or something in the middle: have a
>> test_expectations.txt for mac, mac-leopard, mac-snowleopard, one for qt*,
>> one for all the WebKit2 ports, etc.  I suspect we'll want to go with
>> something in the middle.
> 
> Tony's description is spot-on. The only reason we don't support
> cascading expectations files is because it wasn't clear to me how we
> would want things to work (i.e., which of the choices above) and I
> wasn't able to get much input a few months ago.
> 
> If there is a consensus, it will be easy to implement, so how do we
> actually want this to work?

Out of the different options raised so far, I like the idea of having an include directive. Then ports can decide for themselves how much factoring is appropriate.

I think one giant expectations file for all ports is probably too complicated to be manageable, but an include-based setup would let specific ports share one expectations file for many configurations if they wish.

(Also and incidentally, I'd suggest renaming the file to TestExpectations or TestExpectations.txt to better match WebKit naming style, but this is a much more trivial issue.)

Regards,
Maciej