[webkit-dev] [webkit-changes] [52980] trunk/LayoutTests

[Eric Seidel]

On Fri, Jan 8, 2010 at 12:14 PM, Alexey Proskuryakov <ap at webkit.org> wrote:
>
> It's not clear where the bug is though. If there is an uninitialized memory
> read in a later test, then earlier tests that randomly affect it can be
> perfectly correct. The fact that on other platforms the problem was not
> fixed support this hypothesis to some degree.

Agreed, we don't fully understand this issue, thus things are still
broken.  They're just differently broken than they were before.

We could continue a slightly altered version of the previous
brokenness by checking in failing results.  I encouraged Ossy to skip
them instead because at least with a skipped list we have a single
list of issues to fix.  With checked in failures it's never clear
whether they're failures or not.  I'm happy with either solution if
you'd prefer checking in failing results, that's doable.

Although I think rolling out the run-webkit-tests change is OK, I
don't think it's the best solution.  We shouldn't be afraid of
re-ordering tests.  We just have to make sure we try and discover the
tests which affect other tests and properly skip them as we tried to
do here.

> As I said before, it's good when flaky tests cause pain and impede progress
> (as long as those tests are correct, of course). That's their way of making
> us investigate problems which can quickly turn into exploitable security
> bugs in shipping software otherwise.

I disagree.  I see flaky tests as no value to the project whatsoever.
The build bots are about maintaining a known good state, not about
being a todo list.  Skipped lists and bug lists should be our todo
lists.  A green bot running a few fewer tests is much more valuable
than a bot which is occasionally red but runs all the tests.  WebKit
just needs to make an effort to drive our Skipped list to zero over
time and be less scared to use it to keep the bots green.

-eric