[webkit-dev] Running new/modified tests on EWS bots

[Osztrogonác Csaba]

Hi,

I'd like this idea basically, it would be great
if you or somebody else could work on it.

One of my former collegue started to investigate the same
thing 1.5 years before, but it was lost among many tasks
with higher priorities. Feel free to reuse these bugs and
WIP patches: (or file new ones)
- https://bugs.webkit.org/show_bug.cgi?id=117831
- and bugs in its "depends on" list

In my opinion red tree and flakey tets could cause many
problems, because EWS is designed primarily for green tree.
But unfortunately only Mac (and now Win release) bots are green
almost all the time. I can't remember if EFL or GTK bots were
green in the last 1-2 years ever. :-/

Now the tester EWS bots (Mac WK1 and WK2) run all tests. If all tests
pass, there is no problem. But if there is only one failure, it tries
to build and run tests without the patch. If all tests pass, the patch
caused the regression. If there are exactly the same failures on the
trunk and with the patch, the bubble becomes green.

But flakey failures on the trunk can make EWS crazy. I remember that
EWS wasn't able to process any patch for days because of 1-2 flakey
failures on the trunk. If we enabled full or partial testing on
unstable EWSs, they would become very slow, unstable and we would
lost the very quick and stable builder EWSs. Maybe we should have
separated builder and builder-and-tester EWS bots to avoid this kind
of problems, but we would need more hardware for it.

Anyway it would be great if EFL and GTK maintainers could keep the
tester bots green almost all the time. I know it isn't an easy task
and would need more resource. I remember from the QtWebKit era that
1-2 full-time gardener can be enough to keep the tree green, but
it means only minor buildfixes, expected updates, new bug reports
and skipping/marking new failures time to time. To increase the
stabilility, fix regressions, decrease the number of skipped tests
would need much more resource.

Of course, I willingly help you, but unfortunately I don't have
too much time for this topic nowadays. But feel free to cc me
to related bugs and I'll try to help if I can.

br,
Ossy

youenn fablet írta:
> Hi,
> 
> Related to the webkit contributor meeting discussion related to ports,
> I would find it useful if EWS bots (gtk, efl, win, ios) were running
> the tests that are modified/created by a patch.
> 
> The idea would be to turn yellow the port bubble whenever one of these
> tests do not pass. Results would be uploaded to bugzilla.
> 
> This would give an incentive for patch developers to try fixing the
> tests on these ports.
> That may reduce (slightly? noticeably?) port maintainers gardening effort.
> That would also be valuable when importing test suites.
> 
> Any potential issue? objection to move that forward?
> Anyone willing to help? Thoughts?
> 
> Thanks,
>    Youenn