[webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 9, 2017, at 8:44 AM, youenn fablet <youennf at gmail.com> wrote:
> 
> 
> Besides other issues mentioned, testharness tends to result in more verbose tests compared to js-test, at least for simple cases.
> 
> For synchronous tests, I am not sure there is any big difference one way or the other.
> With asynchronous tests, it might be true, but using testharness.js/promise_test usually improves things there.
>  I personally find it easier to not wrap code-to-be-tested into quotes.

It seems like it's a matter of personal preference to some extent. So I am not sure we should recommend one or the other (besides for tests that are intended to be contributed to web-platform-tests).

> 
> Another concern is the lack of verbose output which reduces the ability to debug failing tests.
> This can be partially fixed by authoring tests with that issue in mind.
> For instance, having a big promise_test to handle the asynchronous aspect of a test and nested test() inside it.

I don't think we've ever had a problem with debugging js-test.js tests when they fail.

> 
>> The thing I specifically asked Youenn to ask is, whether we should
>> place a test inside LayoutTests/wpt like LayoutTests/http/tests when
>> we want to write a test using testharness.js which requires some sort
>> of network code.
>> 
>> Since people have had some opinions about directory structures in the past.
> 
> 
> It seems like we need a few different directories, here are my opinions on them:
> 
> (1) Imported web platform tests that don't need a server
>     Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.
> 
> All WPT tests are expected to run behind the WPT server.
> That is the way tests are authored and tested elsewhere.
> If we have an issue with that, it is best to bring that and fix that directly in WPT.
> I encountered several times small issues due to file:// based origins which makes me think defaulting to http is a safe choice.
> 
> One concern is efficiency. We should study that and improve on that.
> Another concern is the ease of running tests for developers: drag&dropping tests into a browser instead of running a server.
> We can partially accommodate this by rewriting testharness.js/testharnessreport.js urls.
> A significant and growing amount of wpt tests will not behave as expected (other resources loaded, origins, need for specific headers, need for https...)
> 
> (2) Imported web platform tests that do need a server
>     Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe under http/tests/ per point (4)
> 
> I don't think this will work, web-platform-tests is organized in terms of features.
> There is no clear separation between file based compatible tests and http based tests like in WebKit.

If we run all the w3c-imported web platform tests under a web server, then obviously we only need one directory. My understanding is that we don't run them under a server at all. So it seems like one part of this proposal is "run everything under LayoutTests/imported/w3c/ from a web server".

> 
> (3) Custom testharness.js tests that don't need a server 
>     Probably these should just go in their normal topic-specific directories and should not need a special directory
> 
> Right.
> The only case where it might make sense to put such tests in a specific WPT-enabled directory is if the plan is to upstream these tests at some point.
> Such tests could be added in imported/w3c/web-platform-tests directly but this requires coordination with resyncing tests at the moment.
> In a not-too-far-future, I hope such tests would directly be authored in imported/w3c/web-platform-tests.

I think it would be cleaner to have a separate directory of tests intended for import, separate from imported tests. It could be right next to imported/w3c/web-platform-tests. I think mixing tests that are imported from upstream and tests intended for eventual upstreaming is more confusing than helpful.

>  
> (4) Custom testharness.js tests that do need a server
>     Can these just be a subdirectory of http/tests/? We have websocket and ssl/tls tests in there too. Would be nice to not need a separate directory for networking tests that to use a particular test framework.
> 
> I do not have strong feelings there, http/wpt might make sense if it is found easier to understand to everybody.
> I'll update the patch accordingly and will land it sometimes this week if there is no additional feedback.

One question I have is whether web platform tests can run under a regular HTTP server (maybe with appropriate configuration) or do we need something special? Is the WPT server more than just a web server with specific configuration settings?

Regards,
Maciej


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170509/06310db8/attachment.html>