[webkit-dev] Upstreaming from LayoutTests to web-platform-tests, coordinating Blink+WebKit

[Alexey Proskuryakov]

> 17 нояб. 2017 г., в 23:02, youenn fablet <youennf at gmail.com> написал(а):
> 
> Tests are available at https://w3c-test.org <https://w3c-test.org/> which makes it easy to share through any tool supporting hyperlinks.
> A webarchive can also be made so that it is easy to share and probably edit such tests.
> Tools like jsfiddle are also a great way to create/share/edit tests.
> I received several bug reports on bugzilla using it and this proved to be efficient.

I don't think that these are applicable verbatim, but there may be some solution. Let's find it before assuming that we can have it.

In particular, 
- editing a WebArchive is not really feasible;
- jsfiddle is cool, but I can't just copy a WPT test to jsfiddle to share it.

> Let me explain why I think that WebKit tests are often more valuable as regression tests than WPT tests are. We add tests as we fix bugs, so we know that the tests are generally for problems that have a high impact on users and developers - that's because someone actually discovered the problem, and someone prioritized it highly enough to fix. We also know that our tests cover code that is error prone, which is why we had bugs in the first place. Of course, anything can be broken, but certain things are less likely to. Compliance tests written for specs are also valuable, but at some point we need to prioritize which tests to investigate and even to run.
>  
> I don't really see why we should prioritize the tests to run when all of them provide clear value to some WebKit members.

We prioritize which tests to run all the time, there are whole configurations for which we don't run any tests. That's largely because it's not enough to simply run the tests - keeping them in a state where they produce actionable results requires a lot of attention.

The time it takes to run tests definitely matters a lot for WebKit. So far we haven't taken a route like the one Robert mentioned, with a huge amount of hardware running shards of tests. There are multiple reasons to that, one of those being that we are very interested in finding bugs below WebKit too, and having dedicated testers helps with that. When the machine gets into a weird state after a few weeks of uptime, it is much easier to start isolating the problem when we can see which specific queues are hitting it. It is also much easier to reason about data collected by centralized systems, such as crash log collection services.

- Alexey


> I agree that we need to prioritize tests we investigate. There can be a solution inside WPT, like adding WebKit specific metadata so that:
> - WPT contributors would communicate with WebKit members whenever changing such tests
> - WebKit contributors would prioritize WPT-WebKit-metadata failing tests
> 
> That said, if these tests are so beneficial to WebKit, they are potentially very useful to other teams as well.
> And vice-versa, we might find really good WPT tests that show useful crashes and failures in WebKit.
> I am experiencing that nowadays with WPT service worker tests.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20171128/b9784da7/attachment.html>