[webkit-dev] Standard process for exporting new WPT tests?

[Robert Ma]

Current maintainer of Chromium's WPT sync chiming in:

Because of the lack of atomic operations across two separate repos, some
kind of transient inconsistency and race conditions are unavoidable.

FWIW, when Chromium's WPT sync was originally designed, there was concern
about increasing the wait time of landing a Chromium change, which partly
led to our design choice of landing the Chromium change first and then the
GitHub export PR. One important mechanism to avoid losing the Chromium
change (because of being overwritten by the next import) that hasn't been
mentioned is that *we re-apply landed-in-Chromium-but-not-merged-on-Github
patches during import*. Of course, this mechanism caused quite some extra
complexities, but has been working well for us.

The approach being taken by the WebKit community (merging GitHub PRs before
landing WebKit patches) is sound and should be simpler. And I'm very
excited to see more contributions to WPT!

On Fri, May 25, 2018 at 4:16 AM Philip Jägenstedt <foolip at chromium.org>
wrote:

> On Fri, May 25, 2018, 00:52 Ryosuke Niwa <rniwa at webkit.org> wrote:
>
>>
>> On Thu, May 24, 2018 at 3:22 AM, Philip Jägenstedt <foolip at chromium.org>
>> wrote:
>>
>>> On Wed, May 23, 2018, 23:43 youenn fablet <youennf at gmail.com> wrote:
>>>
>>>> Le mer. 23 mai 2018 à 14:11, Frédéric Wang <fwang at igalia.com> a écrit :
>>>>
>>>>> On 23/05/2018 22:50, Ryosuke Niwa wrote:
>>>>> > As we have preciously discussed, we should NEVER commit new tests
>>>>> into
>>>>> > LayoutTests/imported/w3c/web-platform-tests.
>>>>>
>>>>
>>>> Ryosuke, correct me if I am wrong, I think you are pointing out the
>>>> following rule:
>>>> Changes to LayoutTests/imported/w3c/web-platform-tests tests should
>>>> land first in WPT repository, then in WebKit repository.
>>>>
>>>
>>> Oh, that is surprising.
>>>
>>> https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit
>>> export, and https://trac.webkit.org/changeset/231788/webkit did modify
>>> the test in place. Do you mean that the WPT PR was merged first, or should
>>> be in general? Chromium and Gecko do it in the other order, and I'd be
>>> interested to understand the trade-offs of flipping the order.
>>>
>>
>> Yes, it's okay for a WebKit commit to merge the change which got merged
>> into WPT but it's never okay to first commit the test into WebKit and then
>> later upstream it to WPT...
>>
>> Any process like this where changes end up in WebKit trunk via anything
>>> except a full WPT import will mean that divergence is possible.
>>>
>>
>> Precisely to avoid these problems.
>>
>
> I'm saying that there's temporary divergence with the WPT-then-WebKit
> order as well.
>
> Whichever side is merged first, the other side can fail to merge because
> any number of reasons, and something/someone then needs to deal with that.
> (See question in reply to Youenn.)
>
> However, one thing I really like about your order is that it makes it more
> straightforward to make WPT repo failures block browser repo merging. In
> the WPT Travis job, we already detect flakiness for Chrome and Firefox and
> I think we should also detect harness errors, and eventually also flag when
> a test goes from passing everywhere to failing everywhere, which is
> probably a mistake. We haven't really figured out how to make that block
> Chromium's CQ yet, and maybe flipping the order would do it.
>
>> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20180525/088b9d34/attachment.html>