[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

[Eric Seidel]

I do not know the history as to why Chromium removed support for
test_expectations cascading.

Ideally we would have fewer test expectations, not more in the future. :)

On Thu, Jul 7, 2011 at 3:16 AM, Balazs Kelemen <kbalazs at webkit.org> wrote:
> On 07/06/2011 07:24 PM, Eric Seidel wrote:
>>
>> On Wed, Jul 6, 2011 at 10:18 AM, Xan Lopez<xan at gnome.org>  wrote:
>>>
>>> On Wed, Jul 6, 2011 at 6:29 PM, Eric Seidel<eric at webkit.org>  wrote:
>>>>
>>>> NRWT uses both!  It will read in all the port's Skipped files, covert
>>>> them to SKIP text_expectations, and add them to your test_expectations
>>>> file.
>>>>
>>>> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/layout_tests/port/webkit.py#L309
>>>>
>>>> For better or worse, NRWT will error out, if you have duplicates in
>>>> your test_expectations file, including duplicates between your
>>>> test_expectations file and your Skipped lists.
>>>
>>> Right, this is what I meant in another email when I said you are not
>>> supposed to use both. Cannot really see a sane use case for this to be
>>> honest. When I transitioned I basically converted Skipped locally to
>>> the new format, got tons of duplicated errors, figured out what was
>>> going on and deleted then deleted Skipped. Maybe this is done so that
>>> you can leave Skipped as it is and start gradually adding stuff to the
>>> new file?
>>
>> This was done to make it possible to bring up NRWT on Mac over a year
>> ago. :)  I'm happy to look at moving to a different configuration now
>> that the project has (mostly) moved to NRWT.
>
> So long term the best is to move from Skipped to text_expectations. But I
> worry about the lack of the cascading logic. At some point we decided that
> we need it in the old system. Why do we think that we won't need it with
> NRWT? I think the cascading reduce the cost of maintaining the skipped
> lists. WebKit2 is the best example. We have a common skip list that lists
> all the tests that are failing due to a common WebKit2 specific reason. In
> that way, I can skip tests that appearing when I work and Apple folks are
> sleeping and they don't need to worry about that and the same is true in the
> reverse direction.
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>