[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[youenn fablet]

Thanks for taking the time to share this additional information.
I think this is helpful to make progress.
Please see inline for some more comments related to the points you are
bringing to the table.

Stepping back from the WPT server specific issue, being optimistic and
assuming that we are able to run WPT tests with good enough performances.
Migration potential downsides I heard so far:
- Prioritization of tests to investigate might be harder.
- Sharing tests with other teams might be harder if subresource links are
not relative.
These two seem like solvable issues to me.

Le ven. 17 nov. 2017 à 10:09, Alexey Proskuryakov <ap at apple.com> a écrit :

>
> 17 нояб. 2017 г., в 9:18, youenn fablet <youennf at gmail.com> написал(а):
>
>
> Chris recently noticed that some heavily used files (testharness*) were
> cacheable through Apache but not WPT.
> This is now fixed and should improve WPT performances.
>
>
> This is part of <https://bugs.webkit.org/show_bug.cgi?id=178277>, another
> part is that the server is 10x slower than Apache.
>

Other measurements showed 3x slower, which makes it still worthwhile to
explore.
We need to be cautious here though since optimization is all about chasing
where time is actually spent.

If we can prove to ourselves that this is an important issue, we should
discuss with the WPT community to see how to fix this issue.
In addition to better caching, other optimization like
https://github.com/w3c/wptserve/pull/86 may bring some additional benefit.

If we do not find any good solution at WPT server level, we still have the
option to run some tests as file-based.
Ryosuke mentioned the possibility to classify tests at import time by
checking their results when served through WPT server and as file URLs.

I just tested on my MacBook Pro, and WPT tests took 23% of time while being
> only 9% of the total count. Taking in mind that WebKit own tests have
> higher value due to the way we choose what to test (see below), that's not
> a great story in my opinion.
>

These numbers are difficult to interpret.
WPT authoring style is multiple tests in a single file, which is bad for
stability but potentially good for performances.
WebKit usually prefers one file/one test.

If we want to talk to WPT community about this, we need to provide some
more tangible numbers.
We could try to run a large subset of WPT tests that run the same through
Apache and WPT.
I just did that on a small subset of IDB tests. This seems to show
something like a 25% slowdown.

One other thing that we discussed before was the operational complexity of
> running WPT tests. We frequently need to share tests with people who don't
> work on WebKit directly, but have the need to edit and run our tests.
> Inability to drag and drop a local copy into a Safari window is a deterrent
> to addressing problems caught by the tests. I think that the response we
> got was that tests will continue to require a server to run.
>

Tests are available at https://w3c-test.org which makes it easy to share
through any tool supporting hyperlinks.
A webarchive can also be made so that it is easy to share and probably edit
such tests.
Tools like jsfiddle are also a great way to create/share/edit tests.
I received several bug reports on bugzilla using it and this proved to be
efficient.


> Let me explain why I think that WebKit tests are often more valuable as
> regression tests than WPT tests are. We add tests as we fix bugs, so we
> know that the tests are generally for problems that have a high impact on
> users and developers - that's because someone actually discovered the
> problem, and someone prioritized it highly enough to fix. We also know that
> our tests cover code that is error prone, which is why we had bugs in the
> first place. Of course, anything can be broken, but certain things are less
> likely to. Compliance tests written for specs are also valuable, but at
> some point we need to prioritize which tests to investigate and even to run.
>

I don't really see why we should prioritize the tests to run when all of
them provide clear value to some WebKit members.
I agree that we need to prioritize tests we investigate. There can be a
solution inside WPT, like adding WebKit specific metadata so that:
- WPT contributors would communicate with WebKit members whenever changing
such tests
- WebKit contributors would prioritize WPT-WebKit-metadata failing tests

That said, if these tests are so beneficial to WebKit, they are potentially
very useful to other teams as well.
And vice-versa, we might find really good WPT tests that show useful
crashes and failures in WebKit.
I am experiencing that nowadays with WPT service worker tests.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20171118/1f9f7838/attachment.html>