[webkit-dev] Another WPT bite

[Ryosuke Niwa]

Forgot to CC webkit-dev.
- R. Niwa


On Tue, May 9, 2017 at 12:36 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
> On Tue, May 9, 2017 at 1:12 AM, Maciej Stachowiak <mjs at apple.com> wrote:
>>
>>
>> On May 8, 2017, at 11:15 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>
>> On Mon, May 8, 2017 at 11:01 PM, Brady Eidson <beidson at apple.com> wrote:
>>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <beidson at apple.com> wrote:
>>
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>>
>> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>> I just object to making it the "recommended way" of writing tests.
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>>
>> Yes.
>>
>> If not, why?
>>
>> N/A
>>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>>
>> "It's okay to write your test however you'd like. If you were considering
>> using js-test, maybe you should consider using testharness instead."
>> Is that's what's being proposed?
>>
>>
>> On May 8, 2017, at 10:44 PM, Ryosuke Niwa <rniwa at webkit.org> wrote:
>>
>> On Mon, May 8, 2017 at 10:17 PM, Brady Eidson <beidson at apple.com> wrote:
>>
>> But now talking about testharness.js directly, I object on the grounds of "a
>> file:// regression test is dirt easy to hack on and work with, whereas
>> anything that requires me to have an httpd running is a PITA"
>>
>>
>> I think whether we use file:// or http:// is orthogonal point to using
>> testharness.js.  Many of the tests Chris and I have written using
>> testharness.js are checked into regular LayoutTests/ directories, and
>> they work just fine.
>>
>>
>> Yes, I misunderstood this in Youenn's original message. Good to know!
>>
>>
>> I just object to making it the "recommended way" of writing tests.
>>
>>
>> Would you equally object to making js-test.js / js-test-pre.js the
>> recommended way of writing tests?
>>
>>
>> Yes.
>>
>> If not, why?
>>
>>
>> N/A
>>
>> What we're suggesting is to give preferential treatments to
>> testharness.js over js-test.js / js-test-pre.js when you were already
>> planning to write a test with the latter two scripts.
>>
>>
>> "It's okay to write your test however you'd like. If you were considering
>> using js-test, maybe you should consider using testharness instead."
>>
>> Is that's what's being proposed?
>>
>>
>> Besides other issues mentioned, testharness tends to result in more verbose
>> tests compared to js-test, at least for simple cases.
>>
>>
>> The thing I specifically asked Youenn to ask is, whether we should
>> place a test inside LayoutTests/wpt like LayoutTests/http/tests when
>> we want to write a test using testharness.js which requires some sort
>> of network code.
>>
>> Since people have had some opinions about directory structures in the past.
>>
>>
>> It seems like we need a few different directories, here are my opinions on
>> them:
>>
>> (1) Imported web platform tests that don't need a server
>>     Currently LayoutTests/imported/w3c/web-platform-tests, which seems fine.
>> (2) Imported web platform tests that do need a server
>>     Probably should be under LayoutTests/imported/w3c/ somewhere, or maybe
>
> I don't think it's a good idea to split web-platform-tests into two
> separate directories like that because I'd imagine there are resource
> dependencies between tests.  Note that this whole process of detecting
> & separating tests that can be ran locally is the work we'd have to
> do, and upstream won't maintain it.  Given that, we should be making
> the minimal amount of differences to the upstream directory structure
> as well.
>
> Granted, this would make it harder to know which tests require a
> server and which one doesn't. Perhaps this is bad enough that we need
> two directories after all but perhaps there is some other way to solve
> this problem like adding suffix to test names. (Although some ref
> tests in web-platform-tests use other tests as expectation but ref
> tests don't tend to require servers anyway; I know this is a terrible
> idea but we don't really have a control over what upstream does).
>
>> under http/tests/ per point (4)
>> (3) Custom testharness.js tests that don't need a server
>>     Probably these should just go in their normal topic-specific directories
>> and should not need a special directory
>> (4) Custom testharness.js tests that do need a server
>>     Can these just be a subdirectory of http/tests/? We have websocket and
>> ssl/tls tests in there too. Would be nice to not need a separate directory
>> for networking tests that to use a particular test framework.
>
> I guess we could put them under http/tests/wpt/ then?  One weird thing
> is that by default, http/tests/wpt/ would map to the top-level
> directory during testing: i.e. /http/tests/wpt/ would be localhost:X/
> where X is WPT server's port.
>
> - R. Niwa