[webkit-dev] Running new/modified tests on EWS bots

[Carlos Alberto Lopez Perez]

On 19/03/15 16:46, youenn fablet wrote:
> Hi,
> 
> Related to the webkit contributor meeting discussion related to ports,
> I would find it useful if EWS bots (gtk, efl, win, ios) were running
> the tests that are modified/created by a patch.
> 
> The idea would be to turn yellow the port bubble whenever one of these
> tests do not pass. Results would be uploaded to bugzilla.
> 
> This would give an incentive for patch developers to try fixing the
> tests on these ports.
> That may reduce (slightly? noticeably?) port maintainers gardening effort.
> That would also be valuable when importing test suites.
> 
> Any potential issue? objection to move that forward?
> Anyone willing to help? Thoughts?
> 

I think that having an EWS running only the tests that the patch touches
is of limited value because a patch can break unrelated tests.

However running all the tests requires to have the tree green and this
is not possible for GTK or EFL because it will require more manpower
than the currently available.

An idea that comes to mind for running all the tests without requiring
to have the tree green is the following:

1) EWS either run all the tests without the patch or download from
https://build.webkit.org/results the results from the bot for the
revision if they are available.
2) EWS run all the tests with the patch.
3) EWS gets the diff from 1. and 2.
4) EWS runs only the tests that broke from 1 to 2 several times, in
order to discard the ones that are flaky and only report the ones that
on every run fail now.

If I'm not overlooking something, I think that this would allow to
identify what tests a given patch breaks and it won't require to have
the tree green, neither it will be affected by flaky tests.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 883 bytes
Desc: OpenPGP digital signature
URL: <https://lists.webkit.org/pipermail/webkit-dev/attachments/20150319/8d517ee5/attachment.sig>