[webkit-dev] Standard process for exporting new WPT tests?

[Philip Jägenstedt]

On Fri, May 25, 2018 at 6:09 PM youenn fablet <youennf at gmail.com> wrote:

>
> However, one thing I really like about your order is that it makes it more
>> straightforward to make WPT repo failures block browser repo merging. In
>> the WPT Travis job, we already detect flakiness for Chrome and Firefox and
>> I think we should also detect harness errors, and eventually also flag when
>> a test goes from passing everywhere to failing everywhere, which is
>> probably a mistake. We haven't really figured out how to make that block
>> Chromium's CQ yet, and maybe flipping the order would do it.
>>
>
> Exactly, WPT infrastructure has some really nice sanity checks that we do
> not have in WebKit and it sounds really great to benefit from it.
>
> That said, this will put some burden on WPT infrastructure like being fast
> enough to not slow down our own process.
>

If you run into trouble here, please let us all know so that things that
are holding back WebKit contributions don't linger and are forgotten.


> We should also probably run as much as possible the WPT sanity checks
> locally.
> We are now checking WPT linter as you suggested a while back.
> Maybe there are some other WPT checks that could be run in the WebKit
> context.
>

The one that comes to mind is stability checking. We're now running each
new/modified test 10 times in Chrome and Firefox on Travis, and it does
catch flaky tests. We have no straightforward way of doing the same for
Safari, which means no stability checking for Safari. Perhaps we should be
testing with the GTK port, or what is the Linux port that you run on WebKit
bots and want to stay in good shape?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20180531/b8a5dc1d/attachment.html>