[webkit-dev] EWS doesn't lie!

[Osztrogonác Csaba]

On 02/10/2014 09:51 AM, youenn fablet wrote:
> Is it by design that only mac bots run regression tests? Technical
> issue? Lack of resources?

Technically, it's so easy to make an EWS to run layout tests
too with adding a "runTests": true" to the ews.json file:
https://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/ews.json

But in my opinion it wouldn't be a good idea to enable layout tests
on Windows, GTK, EFL EWS bots, because it would make them absolutely
useless and we would lost the information if a patch builds or not.

To have a quite stable and working tester EWS, the buildbot for the
given platform must be green _almost all the time_. If there is at
least one failing test, the testing is at least twice slower, because
the EWS runs the test with the patch and then without the patch to
check if the list of the failing tests are same. Additionally the
given port must be very stable. If there are any small flakiness,
the EWS wouldn't pass ever and would stuck in an infinite loop.

The rough true is that now only the Mac platform is stable and green
enough to have tester EWS bots. (There are ~210 +/-5 failures on the
Windows bots from the cstack merge, ~205 +/-10 failures on EFL-WK1 long
time ago, ~80 +/- 2 failures on GTK-WK1 lone time ago, ~ 60 +/- 5
failures on GTK-WK2, ...)

Additionally to have tester EWS, port maintainers should have to setup
many new hardware (min. 4-8 machines with 4/8 cores per port to have
acceptable runtime) and EWS runtime would be much more slower than
the runtime of build only EWS bots, because bulding + running tests
take ~ an hour everywhere.

Ossy