[webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 13, 2017, at 5:26 PM, Michael[tm] Smith <mike at w3.org> wrote:
> 
> Maciej Stachowiak <mjs at apple.com>, 2017-05-13 14:58 -0700:
>> ... From what I gather, there are a lot of tests where only the paths to
>> the test harness end up requiring the server.
> 
> Yeah that’s the case for the vast majority of tests. Relatively few — less than
> 5% altogether, I’d estimate — actually rely on any special server behavior.
> 
> Of the ones needing special server behavior, I think even there by far most are
> just cases of an accompanying foo.html.headers file in the tree along with the
> foo.html test, to specify that the server send particular response headers.
> 
> Out of ~50,000 test files in WPT, there are less than 1000 with a .headers file.
> 
> I think the next-most-common case that require special server behavior are cases
> where the server is doing some parsing and substitution of special parameters.
> In those cases the test file itself will be named in the pattern foo.sub.html,
> or one of its JS assets in the pattern foo.sub.js.
> 
> Out of ~50,000 test files in WPT, less than 300 are *.sub.* files.
> 
> So those two cases take together amount to only 3% of the total. So even with
> whatever else I’m missing added to that I’d estimate the number of tests that
> don’t rely on special server behavior is on the order of 95%.
> 
> So those ~95% all only need the “/resources/testharness.js” path to the test
> harness to resolve and then they’ll just work.
> 
>> Doing the fixup on import seems bad to me, since it seems safer and
>> cleaner for our WPT checkout to match WPT. But we could follow the
>> practice of using relative URLs for self-created tests, and perhaps not
>> even run them under the server when they don't need it.
>> 
>> For upstream, perhaps we could advocate with WPT to use relative paths to
>> load the harness
> 
> Given that a specific problem case Alexey mentioned was linking to tests within
> the WebKit Trac and having them run as expected, I wonder if at least y’all
> could find a way to just make https://trac.webkit.org/resources/testharness.js
> work — I guess by making it redirect to some place where you have a current WPT
> checkout. That’d at least solve things for the main specific case that’s been
> brought up so far being a real problem.

Alexey mentioned trac because it's something we couldn't easily solve with a command-line tool. The far more common case is engineers loading individual test cases to debug them. As you mentioned ~95% of test cases would load fine as a local file, except for the path where they expect testharness.js.

For the engineer use case, we can make a command-line tool to launch the server and load the test. But it's kind of sad that in ~95% of cases, the only value provided by the server is resolving the path to testharness.js. If tests referenced testharness.js via relative path, then most of the time they could be loaded as local files just fine, which would be more convenient (as well as, I believe, solving the "external trac link" issue).


>> (and perhaps make sure that tests that absolutely require the server fail in a
>> way that clearly indicates this for the tests that truly do need networking or
>> some other facility of the WPT server).
> 
> Yeah that would be good to make happen in general regardless. But I think right
> now that maybe can mostly be (programatically) determined by (1) checking if the
> name of the test file itself is in the *.sub.* pattern, or (2) checking to see
> if there’s an accompanying *.headers file for the test.

It would only be relevant for tests to give better diagnostics if they were set up to run outside the WPT server at all, which they aren't. This suggestion is somewhat conditional on consideration of that other suggestion (which is basically to load required static resources via relative path).

Regards,
Maciej