[webkit-dev] Another WPT bite

[Maciej Stachowiak]

> On May 12, 2017, at 7:49 PM, ap at webkit.org wrote:
> 
> 
>> 12 мая 2017 г., в 19:38, Brian Burg <bburg at apple.com <mailto:bburg at apple.com>> написал(а):
>> 
>>> I think that I explained it very clearly, but let me try again.
>>> 
>>> When there is a test failure that I need to communicate to others, I say something "please open <https://trac.webkit.org/export/216812/webkit/trunk/LayoutTests/fast/images/destroyed-image-load-event.html <https://trac.webkit.org/export/216812/webkit/trunk/LayoutTests/fast/images/border.html>> in Safari to reproduce". That's very easy to do, and makes it very easy for others to work on the issue.
>>> If your test requires complex setup, like WPT does, then I may not have the time to write up complicated steps to reproduce, or the person who gets the bug may not have the time to follow them. Those people don't have a WebKit checkout, so scripts won't help. This makes the test less effective, as problems that it finds are less likely to be addressed.
>> 
>> If the person works on WebKit, then it seems unreasonable that they would do work without a checkout.
> 
> It is correct that people who work on WebKit usually have a checkout. So I was taking about people who don't work on WebKit.
> 
>> If they don’t work on WebKit, then you could run wptserve on a machine somewhere and link to that copy. We have several servers that exist solely to host test content, it doesn’t seem like a big deal to make one of them update regularly and relaunch wptserve to pick up test harness changes.
> 
> Yes, there is a number of things one could do. Those things would work in some cases but not in others - I mentioned linking to a stable version that won't change, which is something that trac gives us for free, and it would be non-trivial to implement otherwise.
> 
> In practice, the best approach would be to reduce the test to a minimum that doesn't use complex harnesses before ending it over. Everyone likes minimal test cases.

I think WPT is becoming enough of a broad community norm that it's not unreasonable to point to a WPT server. It does seem regrettable that tests become dependent on being run through the server even when in principle they could run as local files. From what I gather, there are a lot of tests where only the paths to the test harness end up requiring the server. Doing the fixup on import seems bad to me, since it seems safer and cleaner for our WPT checkout to match WPT. But we could follow the practice of using relative URLs for self-created tests, and perhaps not even run them under the server when they don't need it.

For upstream, perhaps we could advocate with WPT to use relative paths to load the harness (and perhaps make sure that tests that absolutely require the server fail in a way that clearly indicates this for the tests that truly do need networking or some other facility of the WPT server).

Regards,
Maciej

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170513/88dd554b/attachment.html>