[Webkit-unassigned] [Bug 105534] New: Let EWS not cq- patches on test failures for not so stable ports

[bugzilla-daemon at webkit.org]

https://bugs.webkit.org/show_bug.cgi?id=105534

           Summary: Let EWS not cq- patches on test failures for not so
                    stable ports
           Product: WebKit
           Version: 420+
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: Normal
          Priority: P2
         Component: New Bugs
        AssignedTo: webkit-unassigned at lists.webkit.org
        ReportedBy: ossy at webkit.org
                CC: eric at webkit.org, hausmann at webkit.org,
                    tony at chromium.org, dominik.rottsches at intel.com,
                    zandobersek at gmail.com, dpranke at chromium.org,
                    noam at webkit.org, kadam at inf.u-szeged.hu


The idea is from https://bugs.webkit.org/show_bug.cgi?id=104815

Now an EWS bot becomes red and set cq- if it can't process its task without any error. 
The type of the error ( build failure / layout test failure ) is irrelevant.

Maybe we could add "should_CQ-_if_the_build_fails" and "should_CQ-_if_test_fails" properties
(of course with better naming) to the AbstractEarlyWarningSystem class and let each ports 
determine if they are so stable to cq- patches on test failures or not. We can easily make
_post_reject_message_on_bug function cq- patches only if the build/test failed _and_ 
the property mentioned above is set.

---------------------------------------------------------------------------------------------

Otherwise to run tests on EWS bots need many hardware resources and would have bigger
latency then build only EWS bots. That's why it would be better to have very quick
build only EWS bots and many, but slow tester EWS bots.

Let's calculate a little bit: Qt EWS build only bots regularly process ~900 patches a week.
It means that they process a patch on all ~11 minutes. (A typical build is only 1-2 minutes
and a clean build is only 5 minutes thanks to the huge icecc build farm.) But run tests on
a patch takes ~30 minutes (with 1 thread, because running tests on Qt with parallel threads
are very flaky and useless on bots), so we have to have minimum 3-4 new machines to be able 
to add Qt-tester EWS bots. (Or 6-8 if we want to have WK1 and WK2 testers too) 

(I checked the EFL buildbots too, they run WK1 tests in 15 minutes with 8 
threads and WK2 tests in 20 minutes with 12 threads.)

The question is first if Qt/EFL want and have enough hardware resource to run 
tests on EWS bots. If yes, this bug report is valid and should be fixed.

-- 
Configure bugmail: https://bugs.webkit.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.