[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Philip Jägenstedt]

Hi Danyao,

I'm afraid that I don't yet have proper breakdown of this, but this is from
some quick grepping in WebKit:

   - 3200k *-expected.html outside of LayoutTests/imported, that might work
   as reftests.
   - 60k files that aren't called *-expected.*, again outside of
   LayoutTests/imported/
   - 765 of those have "testharness.js" somewhere.
   - 3800 of those have "internals." or "eventSender." somewhere

So, it's certainly not the case that there are tens of thousands of tests
that could easily be upstreamed. And no doubt tens of thousands of tests
that will best be left alone for a long time to come.

But, at the very least I think there will be some hundreds of tests that
could be contributed with some mechanical text transforms that don't
introduce much chance of regressing the tests.

On Wed, Nov 15, 2017 at 5:16 PM Danyao <danyao at chromium.org> wrote:

> Hi Philip,
>
> Ali and I would be willing to help this wherever we can. It seems to us
> that there may be a few different classes of tests that require varying
> degree of effort to migrate. Do you have any estimates on how many tests
> fall into each category, maybe based on your analysis of Blink LayoutTests?
>
>    1. braindead to migrate
>    - 1a) ref tests not using window.internals
>       - 1b) tests using internals API that have trivial equivalent in
>       testharness
>    2. tests using non-trivial internals API so require some domain
>    expertise in the functionality under test
>       - 2a) priority features: maybe active development so more likely to
>       diverge in the near future?
>       - 2b) long tail
>    3. tests that can't be migrated to WPT
>       - 3a) tests that explicit use layerTreeAsText (tests in the
>       compositing layer)
>       - 3b) tests that dump render object tree (testing browser internals)
>
>
> Thanks,
> Danyao
>
> On Wed, Nov 15, 2017 at 5:02 AM, Philip Jägenstedt <foolip at chromium.org>
> wrote:
>
>> Hello webkit-dev! (ecosystem-infra in Bcc)
>>
>> TPAC was last week, and there was much talk about web-platform-tests.
>> Some notes are at
>> https://lists.w3.org/Archives/Public/public-test-infra/2017OctDec/thread.html and
>> in particular https://www.w3.org/2017/11/07-testing-minutes.html.
>>
>> In Blink, we're thinking seriously about what it'd take to upstream large
>> parts of LayoutTests into web-platform-tests, and we've realized that there
>> are some risks here, beyond just making sure the tests are good and per
>> spec. Specifically, one bad outcome would be if Blink successfully
>> upstreamed thousands of tests that are also in WebKit, which then begin to
>> diverge in small and large ways. That'd leave WebKit with more duplication
>> between its own tests and web-platform-tests than any other engine, and a
>> headache that grows over time.
>>
>> So, I think this would require close cooperation with the WebKit project.
>> Some different ways this might happen:
>>
>>    1. A Blink developer and WebKit developer work together at the same
>>    time to move their common tests in some area into web-platform-tests, each
>>    removing identical tests that were upstreamed by the other and
>>    consolidating what remains.
>>    2. A Blink developer works to first upstream tests from WebKit (using
>>    WebKit's coming export mechanism), waits for it to be imported into Blink,
>>    and then removes the tests from Blink.
>>    3. The reverse situation.
>>
>>
>> Are there already some areas where the first approach might be doable,
>> where the Blink and WebKit folks already know each other and are eager to
>> do this? Is LayoutTests/css3/flexbox/ by any chance such a case?
>>
>> I realize it's hard to judge these approaches in the abstract, but am
>> curious to hear any enthusiasm or concerns about it.
>>
>> Thanks!
>>
>> _______________________________________________
>> webkit-dev mailing list
>> webkit-dev at lists.webkit.org
>> https://lists.webkit.org/mailman/listinfo/webkit-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20171117/e70c23c0/attachment.html>