[webkit-dev] Best practices for failing a flaky tests (was Re: Switching to new-run-webkit-tests)

[Adam Barth]

On Wed, Jul 6, 2011 at 9:00 AM, Adam Roben <aroben at apple.com> wrote:
> On Jul 6, 2011, at 11:58 AM, Adam Barth wrote:
>> I'm not sure we've quite figured that out yet.  NRWT supports both
>> Skipped lists and test_expectations.txt, which is a more expressive
>> (but also more complex) version of Skipped lists.  IMHO, we should
>> wait for the dust to settle on the transition before changing our
>> practices.
>
> OK. Then I have another question:
>
> What should I do to make the Leopard and SnowLeopard bots green, now that they have switched to NRWT?

Looking at Leopard, there's one jscore-test issue and two
run-webkit-test issues:

http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r90460%20(33799)/results.html

One of the run-webkit-test issues seems to relate to -0.  It's
possible that's a test harness issue, but it seems more likely that
it's a regression in JavaScriptCore.  The other is some SVG foreign
object issue, which I have less insight into.  We can either fix the
regressions, update the expected.txt files, or add the test to the
Skipped list, as before.

On SnowLeopard, there are also two failing tests, but I believe these
tests are related to the NRWT transition:

http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r90458%20(31133)/results.html

The third-party-cookie-relaxing.html test probably needs to be changed
because it depends on the persistent state of the cookie jar.  Either
Eric or I will dig into the test to figure out how to make it more
robust.

The storagetracker tests also have similar issues.  They appear to be
flaky with NRWT, which is
<https://bugs.webkit.org/show_bug.cgi?id=57799>.  These tests are also
flaky under ORWT (but less so):

https://bugs.webkit.org/show_bug.cgi?id=58835
https://bugs.webkit.org/show_bug.cgi?id=58836

We need to fix both issues, but they didn't seem like issues that
should block the change to NRWT.

That's a somewhat round-about way of not answering your question.  :)

Adam