[webkit-dev] Gated trunk, experiences from OpenStack

[Dirk Pranke]

On Tue, Feb 5, 2013 at 9:46 AM, Martin Robinson <mrobinson at webkit.org> wrote:
> On Tue, Feb 5, 2013 at 9:28 AM, Adam Barth <abarth at webkit.org> wrote:
>> Do you know how they got rid of flakiness in their tests?  We've spent
>> a bunch of effort fixing flaky tests (and in marking the remaining
>> flaky tests as flaky), but there's still a long tail of flakiness.  I
>> wonder if that sort of thing might be different for OpenStack if they
>> have a different approach to testing than we do.
>
> Another useful thing is to know the number of tests in OpenStack.
> WebKit has more tests than any other project I've worked on.
>

There are two other related aspects that make our tests flaky:

1) They're very high level integration tests (mostly), which, as they
cover large swaths of code in each test, are much more susceptible to
flakiness than method-level unit tests.

2) They weren't generally written to be run in parallel, and thus we
often have to be concerned with system-level resource contention.

-- Dirk