[webkit-dev] Another WPT bite

[Michael[tm] Smith]

Maciej Stachowiak <mjs at apple.com>, 2017-05-13 14:58 -0700:
> ... From what I gather, there are a lot of tests where only the paths to
> the test harness end up requiring the server.

Yeah that’s the case for the vast majority of tests. Relatively few — less than
5% altogether, I’d estimate — actually rely on any special server behavior.

Of the ones needing special server behavior, I think even there by far most are
just cases of an accompanying foo.html.headers file in the tree along with the
foo.html test, to specify that the server send particular response headers.

Out of ~50,000 test files in WPT, there are less than 1000 with a .headers file.

I think the next-most-common case that require special server behavior are cases
where the server is doing some parsing and substitution of special parameters.
In those cases the test file itself will be named in the pattern foo.sub.html,
or one of its JS assets in the pattern foo.sub.js.

Out of ~50,000 test files in WPT, less than 300 are *.sub.* files.

So those two cases take together amount to only 3% of the total. So even with
whatever else I’m missing added to that I’d estimate the number of tests that
don’t rely on special server behavior is on the order of 95%.

So those ~95% all only need the “/resources/testharness.js” path to the test
harness to resolve and then they’ll just work.

> Doing the fixup on import seems bad to me, since it seems safer and
> cleaner for our WPT checkout to match WPT. But we could follow the
> practice of using relative URLs for self-created tests, and perhaps not
> even run them under the server when they don't need it.
> 
> For upstream, perhaps we could advocate with WPT to use relative paths to
> load the harness

Given that a specific problem case Alexey mentioned was linking to tests within
the WebKit Trac and having them run as expected, I wonder if at least y’all
could find a way to just make https://trac.webkit.org/resources/testharness.js
work — I guess by making it redirect to some place where you have a current WPT
checkout. That’d at least solve things for the main specific case that’s been
brought up so far being a real problem.

> (and perhaps make sure that tests that absolutely require the server fail in a
> way that clearly indicates this for the tests that truly do need networking or
> some other facility of the WPT server).

Yeah that would be good to make happen in general regardless. But I think right
now that maybe can mostly be (programatically) determined by (1) checking if the
name of the test file itself is in the *.sub.* pattern, or (2) checking to see
if there’s an accompanying *.headers file for the test.

  —Mike

-- 
Michael[tm] Smith https://people.w3.org/mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170514/824ef5a6/attachment.bin>