[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

[Balazs Kelemen]

On 07/06/2011 07:24 PM, Eric Seidel wrote:
> On Wed, Jul 6, 2011 at 10:18 AM, Xan Lopez<xan at gnome.org>  wrote:
>> On Wed, Jul 6, 2011 at 6:29 PM, Eric Seidel<eric at webkit.org>  wrote:
>>> NRWT uses both!  It will read in all the port's Skipped files, covert
>>> them to SKIP text_expectations, and add them to your test_expectations
>>> file.
>>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/webkit.py#L309
>>>
>>> For better or worse, NRWT will error out, if you have duplicates in
>>> your test_expectations file, including duplicates between your
>>> test_expectations file and your Skipped lists.
>> Right, this is what I meant in another email when I said you are not
>> supposed to use both. Cannot really see a sane use case for this to be
>> honest. When I transitioned I basically converted Skipped locally to
>> the new format, got tons of duplicated errors, figured out what was
>> going on and deleted then deleted Skipped. Maybe this is done so that
>> you can leave Skipped as it is and start gradually adding stuff to the
>> new file?
> This was done to make it possible to bring up NRWT on Mac over a year
> ago. :)  I'm happy to look at moving to a different configuration now
> that the project has (mostly) moved to NRWT.
So long term the best is to move from Skipped to text_expectations. But 
I worry about the lack of the cascading logic. At some point we decided 
that we need it in the old system. Why do we think that we won't need it 
with NRWT? I think the cascading reduce the cost of maintaining the 
skipped lists. WebKit2 is the best example. We have a common skip list 
that lists all the tests that are failing due to a common WebKit2 
specific reason. In that way, I can skip tests that appearing when I 
work and Apple folks are sleeping and they don't need to worry about 
that and the same is true in the reverse direction.