[webkit-dev] Where do we put WPT tests to be exported

[Maciej Stachowiak]

> On May 15, 2017, at 9:08 PM, youenn fablet <youennf at gmail.com> wrote:
> 
> I see two main cases:
> - Writer of the patch is making sure to upstream WPT test changes at WebKit landing time. It is ok to make the changes directly in LayoutTests/imported/w3c/web-platform-tests/
> - Writer plans to upstream WPT test changes at some point but wants more time. It is better to develop the tests in LayoutTests/http/wpt and then migrate them later on.

My proposal was different from either of these, it was to have a directory specifically for tests meant to be upstreamed (LayoutTests/http/wpt should contain only tests not meant

I think adding new tests directly to LayoutTests/imported/w3c/web-platform-tests/ is needlessly messy. Most stuff in the imported/ directory is an exact copy of an upstream test suite, so if you run only a specific directory of tests, you know you are running an official conformance suite. With this proposal, it might also contain random tests that will hopefully be upstreamed but maybe not, or might be changed before the PR lands upstream, or might get renamed, or whatever. There's no guarantee that updating from the official version will ever fully resolve the delta. 

I think it would be more elegant to have a parallel directory (LayoutTests/for-export/w3c/web-platform-tests). Then when something is actually upstreamed and then pulled back down, we could delete the staged version. Directly modifying our local copy seems like it could easily lead to long-term divergences slipping through the cracks.

Of course, people could always go run the official copy from w3c-test.org <http://w3c-test.org/> . But we usually leave imported conformance test suites unmodified except the minimum necessary to make them run.

> I would start with an experimental phase with some of us making direct changes in LayoutTests/imported/w3c/web-platform-tests/.
> When we are happy with the tools and think the risk for issues is low enough (or when the bots can handle most of it for us), hacking LayoutTests/imported/w3c/web-platform-tests/ could be the default.

I think the problem with this plan is not tools risk, it's the fact that a directory of imported tests can no longer be trusted to actually just be imported tests. So a smaller number of people doing it to start would not address my concern. It might be that other people don't care about this. My opinion counts no more than anyone else's, and I'd be interested in hearing from others.

Regards,
Maciej

> 
> Le lun. 15 mai 2017 à 21:02, Ryosuke Niwa <rniwa at webkit.org <mailto:rniwa at webkit.org>> a écrit :
> Hi all,
> 
> Youenn is working on a patch to automatically create a GitHub PR to
> export tests from a WebKit patch which includes changes to
> web-platform-tests [1].
> 
> That raises a question as to where we should put new tests or modified
> tests intended to be exported to web-platform-tests from WebKit.
> 
> I think the most obvious option is to use
> LayoutTests/imported/w3c/web-platform-tests/.  However, in the other
> thread about adopting testharness.js (titled Another WPT bite), Maciej
> briefly expressed the preference for creating a new directory:
> https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html <https://lists.webkit.org/pipermail/webkit-dev/2017-May/029022.html>
> 
> Do other people have strong opinions about this?
> 
> - R. Niwa
> 
> [1] https://bugs.webkit.org/show_bug.cgi?id=169462 <https://bugs.webkit.org/show_bug.cgi?id=169462>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org <mailto:webkit-dev at lists.webkit.org>
> https://lists.webkit.org/mailman/listinfo/webkit-dev <https://lists.webkit.org/mailman/listinfo/webkit-dev>
> _______________________________________________
> webkit-dev mailing list
> webkit-dev at lists.webkit.org
> https://lists.webkit.org/mailman/listinfo/webkit-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.webkit.org/pipermail/webkit-dev/attachments/20170515/11f83086/attachment.html>