[webkit-dev] Running new/modified tests on EWS bots

[Gyuyoung Kim]

Hi,

Though I really like to support to run tests on EWS like Mac port, it can
often block to land a patch if tests on EFL or GTK aren't green. So I think
we need to find methods which don't annoy patch owner or reviewer. In this
light, if we're able to check if there is different between last test
result and new test result with a patch as Carlos suggested, it seems to be
more realistic approach. If someone tries to support it, I will also help
to try it If I can.

Gyuyoung.


On Fri, Mar 20, 2015 at 2:41 AM, Carlos Alberto Lopez Perez <
clopez at igalia.com> wrote:

> On 19/03/15 16:46, youenn fablet wrote:
> > Hi,
> >
> > Related to the webkit contributor meeting discussion related to ports,
> > I would find it useful if EWS bots (gtk, efl, win, ios) were running
> > the tests that are modified/created by a patch.
> >
> > The idea would be to turn yellow the port bubble whenever one of these
> > tests do not pass. Results would be uploaded to bugzilla.
> >
> > This would give an incentive for patch developers to try fixing the
> > tests on these ports.
> > That may reduce (slightly? noticeably?) port maintainers gardening
> effort.
> > That would also be valuable when importing test suites.
> >
> > Any potential issue? objection to move that forward?
> > Anyone willing to help? Thoughts?
> >
>
> I think that having an EWS running only the tests that the patch touches
> is of limited value because a patch can break unrelated tests.
>
> However running all the tests requires to have the tree green and this
> is not possible for GTK or EFL because it will require more manpower
> than the currently available.
>
> An idea that comes to mind for running all the tests without requiring
> to have the tree green is the following:
>
> 1) EWS either run all the tests without the patch or download from
> https://build.webkit.org/results the results from the bot for the
> revision if they are available.
> 2) EWS run all the tests with the patch.
> 3) EWS gets the diff from 1. and 2.
> 4) EWS runs only the tests that broke from 1 to 2 several times, in
> order to discard the ones that are flaky and only report the ones that
> on every run fail now.
>
> If I'm not overlooking something, I think that this would allow to
> identify what tests a given patch breaks and it won't require to have
> the tree green, neither it will be affected by flaky tests.
>
>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20150321/2cf16a74/attachment.html>