[Webkit-unassigned] [Bug 108288] On EWS bots, run tests on patch once instead of twice if some tests fail.

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=108288





--- Comment #12 from Dirk Pranke <dpranke at chromium.org>  2013-01-30 17:30:08 PST ---
(In reply to comment #11)
> What I want to achieve with this patch is to simplify the logic for running tests and increase the maximum throughput of the EWS bots (since the tests that behave as expected only get run once with a submitted patch). As you mention, we can also reduce the number of flaky tests reported as failures with negligible runtime penalties (although I kept the number of total runs the same here).
>

I've unfortunately forgotten the details of how EWS views and treats flaky tests and how that's different from how NRWT and the flakiness dashboard view them, and so it's hard for me to respond.

In my ideal world, all of the currently flaky (layout) tests would be marked as such in TestExpectations files, and EWS would not try to have a separate concept of flakiness that it was filtering out. IIRC, We originally didn't do this at least because EWS was designed to work with old-run-webkit-tests, which didn't support marking tests as flaky. And, of course, you need some sort of equivalent concept for flaky c++  tests if you want to worry about them. 

I wouldn't worry about flaky python tests, we don't have a big enough problem there.

> My main idea here, though, is to set us up for being better at actually reporting flaky tests by seeing what NRWT outputs re. flakiness (dependent on getting the FlakyTestReporter working and replying to Eric's comment #8) -- note for example that if a test flakes with a small probability p, the current strategy would need about 1/p^2 tests to detect it as flaky, while the new strategy only needs about 1/p tests. It seems too wasteful to have the flakiness information from NRWT which we ignore.

I'm not sure what you mean by "actually reporting flaky tests". Reporting where? to EWS? to the flakiness dashboard? somewhere else?

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.