[webkit-dev] Skipping Flakey Tests

[David Levin]

On Mon, Sep 28, 2009 at 5:01 PM, Maciej Stachowiak <mjs at apple.com> wrote:

>
> On Sep 28, 2009, at 4:47 PM, David Levin wrote:
>
> I don't believe that the test was checked in a flaky state. It was solid
> for a long time and then something happened...
>
>
> What's "the test" in this context? The network / credentials one?
>

Yep.


>
>
> I'll try to add more logging to this test this evening (after my turn at
> helping chromium stay up to date with WebKit is over).
>
> I'll ping Drew about the other test.
>
>
> It sounds like that test was always buggy, if Drew is right about the
> cause.
>
>  - Macie
>
>
> Dave
>
>
> On Mon, Sep 28, 2009 at 4:40 PM, Eric Seidel <eric at webkit.org> wrote:
>
>> On Fri, Sep 25, 2009 at 2:42 PM, Maciej Stachowiak <mjs at apple.com> wrote:
>> > I like Dave Levin's idea that the first action should be to instrument
>> the
>> > tests so we can find out why they intermittently fail. Especially if the
>> > failure is reproducible on the bots but not on developer systems.
>>
>> I like this idea too.  I don't like the reality that flakey tests are
>> a burden on all developers caused by one.
>>
>> > Using the
>> > skip list should be a last resort, because that hides the failure
>> instead of
>> > helping us diangose the cause.
>>
>> I (respectfully) disagree.  I think we shouldn't be so afraid to skip
>> tests.  We don't allow people to check in compiles which fail.  We
>> don't allow people to check in tests which fail on other platforms
>> (without skipping them) or on every other run.  Why should we allow
>> people to check in tests which fail every 10 runs?  Or worse, why
>> should we leave a known flakey test checked in/un-attended which fails
>> every 10 runs?
>>
>> If we can't easily roll-out the failing tests (or the commit which
>> cause them to start failing), we should skip them to keep the bots (a
>> shared resource) green, so as not to block other work on the project.
>> No?
>>
>> I very much like WebKit's "everyone is responsible for the whole
>> project" culture, but I disagree that the burden of diagnosis should
>> be on the person trying to make a completely unrelated checkin (as is
>> the case when we leave flakey tests enabled in the tree).
>>
>> -eric
>>
>> p.s. I now have two "skipping flakey tests" changes up for review:
>> https://bugs.webkit.org/show_bug.cgi?id=29322
>> https://bugs.webkit.org/show_bug.cgi?id=29344
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20090928/fb1e5e2c/attachment.html>