[Webkit-unassigned] [Bug 108288] On EWS bots, run tests on patch once instead of twice if some tests fail.

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=108288





--- Comment #13 from Timothy Loh <timloh at chromium.org>  2013-01-30 21:23:05 PST ---
(In reply to comment #12)
> I've unfortunately forgotten the details of how EWS views and treats flaky tests and how that's different from how NRWT and the flakiness dashboard view them, and so it's hard for me to respond.

The EWS currently re-runs all the tests when any of them fail (twice). The tests that fail on the first NRWT run and pass on the second run (and vice versa, but we're ignoring these at the moment) are considered flaky. The EWS currently lets a patch pass if one of the two runs is clean, or the results are the same and are to be expected given the failures from the clean tree. If the two runs fail on different tests, we grab a new patch and leave the current one in the queue (this leads to the cycling behaviour we sometimes see, that we should probably get rid of).

The flakiness dashboard looks like it actually uses the flakiness output from NRWT. I think it takes the worst run (e.g. crash > text > pass) for the colour you see.

> In my ideal world, all of the currently flaky (layout) tests would be marked as such in TestExpectations files, and EWS would not try to have a separate concept of flakiness that it was filtering out. IIRC, We originally didn't do this at least because EWS was designed to work with old-run-webkit-tests, which didn't support marking tests as flaky. And, of course, you need some sort of equivalent concept for flaky c++  tests if you want to worry about them. 
>
> I'm not sure what you mean by "actually reporting flaky tests". Reporting where? to EWS? to the flakiness dashboard? somewhere else?

The commit-queue used to report flaky tests via the FlakyTestReporter (see the master - Bug 50856). This isn't working at the moment for as yet unknown reasons, but what I'd want is for it post patches to annotate the TestExpectation files for us (the CQ doesn't re-run tests if the EWS already ran so we'd need to pass the info along). What I'd also like to happen is for the FlakyTestReporter to post a message on your patch if it finds new flaky tests, but we'd need the existing flakes to be marked in TestExpectations before this can happen (otherwise it'll get turned off :-(  ).

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.