[webkit-dev] Skipping Flakey Tests

[Darin Adler]

On Dec 22, 2009, at 4:27 PM, Dirk Pranke wrote:

> In the completely generic case, I hope we are not checking in incorrect results.

We do intentionally check in incorrect results, fairly often. For example, we’ve checked in whole test suites and then generated expected results without studying the tests to see which ones are successful and which are failures.

> An alternative would be to move to the more general syntax (and hopefully, just move to the tool) that Chromium uses.

I’m surprised that Chromium developed a separate tool. If instead the Chromium team had enhanced the WebKit project’s shared run-webkit-tests we’d be better off. How did we end up with two separate tools?!

> Second, there's the question of whether or not you want to track what the "expected incorrect" results are, separate from what the "expected correct" results are. That way, you can detect when a test fails *differently* than it has been in the past.

I do think we want to track this. It’s part of why the original system worked they way it did when I created it back in 2005.

Also, in some cases it may be difficult to generate “correct” results if the engine doesn’t yet have correct behavior at the time the test is being created.

    -- Darin