<div dir="ltr"><div class="gmail_extra">Hi,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Though I really like to support to run tests on EWS like Mac port, it can often block to land a patch if tests on EFL or GTK aren't green. So I think we need to find methods which don't annoy patch owner or reviewer. In this light, if we're able to check if there is different between last test result and new test result with a patch as Carlos suggested, it seems to be more realistic approach. If someone tries to support it, I will also help to try it If I can.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Gyuyoung.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 20, 2015 at 2:41 AM, Carlos Alberto Lopez Perez <span dir="ltr"><<a href="mailto:clopez@igalia.com" target="_blank">clopez@igalia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">On 19/03/15 16:46, youenn fablet wrote:<br>
> Hi,<br>
><br>
> Related to the webkit contributor meeting discussion related to ports,<br>
> I would find it useful if EWS bots (gtk, efl, win, ios) were running<br>
> the tests that are modified/created by a patch.<br>
><br>
> The idea would be to turn yellow the port bubble whenever one of these<br>
> tests do not pass. Results would be uploaded to bugzilla.<br>
><br>
> This would give an incentive for patch developers to try fixing the<br>
> tests on these ports.<br>
> That may reduce (slightly? noticeably?) port maintainers gardening effort.<br>
> That would also be valuable when importing test suites.<br>
><br>
> Any potential issue? objection to move that forward?<br>
> Anyone willing to help? Thoughts?<br>
><br>
<br>
</span>I think that having an EWS running only the tests that the patch touches<br>
is of limited value because a patch can break unrelated tests.<br>
<br>
However running all the tests requires to have the tree green and this<br>
is not possible for GTK or EFL because it will require more manpower<br>
than the currently available.<br>
<br>
An idea that comes to mind for running all the tests without requiring<br>
to have the tree green is the following:<br>
<br>
1) EWS either run all the tests without the patch or download from<br>
<a href="https://build.webkit.org/results" target="_blank">https://build.webkit.org/results</a> the results from the bot for the<br>
revision if they are available.<br>
2) EWS run all the tests with the patch.<br>
3) EWS gets the diff from 1. and 2.<br>
4) EWS runs only the tests that broke from 1 to 2 several times, in<br>
order to discard the ones that are flaky and only report the ones that<br>
on every run fail now.<br>
<br>
If I'm not overlooking something, I think that this would allow to<br>
identify what tests a given patch breaks and it won't require to have<br>
the tree green, neither it will be affected by flaky tests.<br>
<br>
<br>_______________________________________________<br>
webkit-dev mailing list<br>
<a href="mailto:webkit-dev@lists.webkit.org">webkit-dev@lists.webkit.org</a><br>
<a href="https://lists.webkit.org/mailman/listinfo/webkit-dev" target="_blank">https://lists.webkit.org/mailman/listinfo/webkit-dev</a><br>
<br></blockquote></div><br></div></div>