[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[youenn fablet]

I filed https://github.com/w3c/web-platform-tests/issues/8391 for the WPT
server performances issues.
As of upstreaming the tests, would the following solve identified issues
and keep the upstream process lightweight enough?
- Meta tag each upstreamed test to identify this is an upstreamed version
of a WebKit test (and maybe which WebKit test).
- Keep relative paths for upstreamed tests.
    y

Le ven. 17 nov. 2017 à 23:02, youenn fablet <youennf at gmail.com> a écrit :

> Thanks for taking the time to share this additional information.
> I think this is helpful to make progress.
> Please see inline for some more comments related to the points you are
> bringing to the table.
>
> Stepping back from the WPT server specific issue, being optimistic and
> assuming that we are able to run WPT tests with good enough performances.
> Migration potential downsides I heard so far:
> - Prioritization of tests to investigate might be harder.
> - Sharing tests with other teams might be harder if subresource links are
> not relative.
> These two seem like solvable issues to me.
>
> Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov <ap at apple.com> a écrit :
>
>>
>> 17 нояб. 2017 г., в 9:18, youenn fablet <youennf at gmail.com> написал(а):
>>
>>
>> Chris recently noticed that some heavily used files (testharness*) were
>> cacheable through Apache but not WPT.
>> This is now fixed and should improve WPT performances.
>>
>>
>> This is part of <https://bugs.webkit.org/show_bug.cgi?id=178277>,
>> another part is that the server is 10x slower than Apache.
>>
>
> Other measurements showed 3x slower, which makes it still worthwhile to
> explore.
> We need to be cautious here though since optimization is all about chasing
> where time is actually spent.
>
> If we can prove to ourselves that this is an important issue, we should
> discuss with the WPT community to see how to fix this issue.
> In addition to better caching, other optimization like
> https://github.com/w3c/wptserve/pull/86 may bring some additional benefit.
>
> If we do not find any good solution at WPT server level, we still have the
> option to run some tests as file-based.
> Ryosuke mentioned the possibility to classify tests at import time by
> checking their results when served through WPT server and as file URLs.
>
> I just tested on my MacBook Pro, and WPT tests took 23% of time while
>> being only 9% of the total count. Taking in mind that WebKit own tests have
>> higher value due to the way we choose what to test (see below), that's not
>> a great story in my opinion.
>>
>
> These numbers are difficult to interpret.
> WPT authoring style is multiple tests in a single file, which is bad for
> stability but potentially good for performances.
> WebKit usually prefers one file/one test.
>
> If we want to talk to WPT community about this, we need to provide some
> more tangible numbers.
> We could try to run a large subset of WPT tests that run the same through
> Apache and WPT.
> I just did that on a small subset of IDB tests. This seems to show
> something like a 25% slowdown.
>
> One other thing that we discussed before was the operational complexity of
>> running WPT tests. We frequently need to share tests with people who don't
>> work on WebKit directly, but have the need to edit and run our tests.
>> Inability to drag and drop a local copy into a Safari window is a deterrent
>> to addressing problems caught by the tests. I think that the response we
>> got was that tests will continue to require a server to run.
>>
>
> Tests are available at https://w3c-test.org which makes it easy to share
> through any tool supporting hyperlinks.
> A webarchive can also be made so that it is easy to share and probably
> edit such tests.
> Tools like jsfiddle are also a great way to create/share/edit tests.
> I received several bug reports on bugzilla using it and this proved to be
> efficient.
>
>
>> Let me explain why I think that WebKit tests are often more valuable as
>> regression tests than WPT tests are. We add tests as we fix bugs, so we
>> know that the tests are generally for problems that have a high impact on
>> users and developers - that's because someone actually discovered the
>> problem, and someone prioritized it highly enough to fix. We also know that
>> our tests cover code that is error prone, which is why we had bugs in the
>> first place. Of course, anything can be broken, but certain things are less
>> likely to. Compliance tests written for specs are also valuable, but at
>> some point we need to prioritize which tests to investigate and even to run.
>>
>
> I don't really see why we should prioritize the tests to run when all of
> them provide clear value to some WebKit members.
> I agree that we need to prioritize tests we investigate. There can be a
> solution inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with WebKit members whenever changing
> such tests
> - WebKit contributors would prioritize WPT-WebKit-metadata failing tests
>
> That said, if these tests are so beneficial to WebKit, they are
> potentially very useful to other teams as well.
> And vice-versa, we might find really good WPT tests that show useful
> crashes and failures in WebKit.
> I am experiencing that nowadays with WPT service worker tests.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20171121/ddc4ca26/attachment.html>