[webkit-dev] EWS doesn't lie!

[youenn fablet]

All of this makes sense, but the downsides is that it does not ensure
convergence between the ports.
If this situation is expected to stay for some time, I wonder whether some
middle ground may be found for ports that are build-stable.

For instance, having status information on new/modified tests for all ports
would be useful and not that expensive.
When tests do not pass on some ports, updating the test expectations (and
creating bug entries) would be an improvement over the current situation.


2014-02-10 10:25 GMT+01:00 Osztrogonác Csaba <oszi at inf.u-szeged.hu>:

> On 02/10/2014 09:51 AM, youenn fablet wrote:
>
>> Is it by design that only mac bots run regression tests? Technical
>> issue? Lack of resources?
>>
>
> Technically, it's so easy to make an EWS to run layout tests
> too with adding a "runTests": true" to the ews.json file:
> https://trac.webkit.org/browser/trunk/Tools/Scripts/
> webkitpy/common/config/ews.json
>
> But in my opinion it wouldn't be a good idea to enable layout tests
> on Windows, GTK, EFL EWS bots, because it would make them absolutely
> useless and we would lost the information if a patch builds or not.
>
> To have a quite stable and working tester EWS, the buildbot for the
> given platform must be green _almost all the time_. If there is at
> least one failing test, the testing is at least twice slower, because
> the EWS runs the test with the patch and then without the patch to
> check if the list of the failing tests are same. Additionally the
> given port must be very stable. If there are any small flakiness,
> the EWS wouldn't pass ever and would stuck in an infinite loop.
>
> The rough true is that now only the Mac platform is stable and green
> enough to have tester EWS bots. (There are ~210 +/-5 failures on the
> Windows bots from the cstack merge, ~205 +/-10 failures on EFL-WK1 long
> time ago, ~80 +/- 2 failures on GTK-WK1 lone time ago, ~ 60 +/- 5
> failures on GTK-WK2, ...)
>
> Additionally to have tester EWS, port maintainers should have to setup
> many new hardware (min. 4-8 machines with 4/8 cores per port to have
> acceptable runtime) and EWS runtime would be much more slower than
> the runtime of build only EWS bots, because bulding + running tests
> take ~ an hour everywhere.
>
> Ossy
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20140210/6ddbac6d/attachment.html>