[webkit-dev] Standard process for exporting new WPT tests?

[Philip Jägenstedt]

On Fri, May 25, 2018, 00:52 Ryosuke Niwa <rniwa at webkit.org> wrote:

>
> On Thu, May 24, 2018 at 3:22 AM, Philip Jägenstedt <foolip at chromium.org>
> wrote:
>
>> On Wed, May 23, 2018, 23:43 youenn fablet <youennf at gmail.com> wrote:
>>
>>> Le mer. 23 mai 2018 à 14:11, Frédéric Wang <fwang at igalia.com> a écrit :
>>>
>>>> On 23/05/2018 22:50, Ryosuke Niwa wrote:
>>>> > As we have preciously discussed, we should NEVER commit new tests into
>>>> > LayoutTests/imported/w3c/web-platform-tests.
>>>>
>>>
>>> Ryosuke, correct me if I am wrong, I think you are pointing out the
>>> following rule:
>>> Changes to LayoutTests/imported/w3c/web-platform-tests tests should land
>>> first in WPT repository, then in WebKit repository.
>>>
>>
>> Oh, that is surprising.
>>
>> https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit
>> export, and https://trac.webkit.org/changeset/231788/webkit did modify
>> the test in place. Do you mean that the WPT PR was merged first, or should
>> be in general? Chromium and Gecko do it in the other order, and I'd be
>> interested to understand the trade-offs of flipping the order.
>>
>
> Yes, it's okay for a WebKit commit to merge the change which got merged
> into WPT but it's never okay to first commit the test into WebKit and then
> later upstream it to WPT...
>
> Any process like this where changes end up in WebKit trunk via anything
>> except a full WPT import will mean that divergence is possible.
>>
>
> Precisely to avoid these problems.
>

I'm saying that there's temporary divergence with the WPT-then-WebKit order
as well.

Whichever side is merged first, the other side can fail to merge because
any number of reasons, and something/someone then needs to deal with that.
(See question in reply to Youenn.)

However, one thing I really like about your order is that it makes it more
straightforward to make WPT repo failures block browser repo merging. In
the WPT Travis job, we already detect flakiness for Chrome and Firefox and
I think we should also detect harness errors, and eventually also flag when
a test goes from passing everywhere to failing everywhere, which is
probably a mistake. We haven't really figured out how to make that block
Chromium's CQ yet, and maybe flipping the order would do it.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20180525/9d503c79/attachment.html>